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JOINT STRIKE FIGHTER ACQUISITION 
REFORM: WILL IT FLY? 


WEDNESDAY, MAY 10, 2000 

House of Representatives, 
Subcommittee on National Security, Veterans 
Affairs, and International Relations, 

Committee on Government Reform, 

Washington, DC. 

The subcommittee met, pursuant to notice, at 10 a.m., in room 
2247, Rayburn House Office Building, Hon. Christopher Shays 
(chairman of the subcommittee) presiding. 

Present: Representatives Shays, Terry, Biggert, and Blagojevich. 

Staff present: Lawrence J. Halloran, staff director and counsel; 
J. Vincent Chase, chief investigator; Robert Newman, professional 
staff member; Jason M. Chung, clerk; David Rapallo, minority 
counsel; and Earley Green, minority assistant clerk. 

Mr. Shays. I would like to call this hearing to order, to welcome 
our witnesses and our guests and again to invite anyone who wants 
to, to take off their coats. It’s a hot room and we have asked it to 
be cooled down, but feel free to take off your jackets. 

The military procurement holiday is about to end. Defense budg- 
ets being debated today on both sides of the Capitol reflect a bi- 
cameral and bipartisan consensus on the need to modernize the 
aging planes, ships, weapons and equipment used to win the cold 
war. Today we discuss the need to modernize the acquisition sys- 
tems the Department of Defense [DOD], will use to procure post- 
cold war weapons systems. 

Just as the weaponry of the last century won’t win the peace in 
the next, the acquisition practices of the past will not efficiently or 
affordably meet future defense needs. Fifteen-year development cy- 
cles enshrine old technologies now rendered obsolete in 15 months. 
Massive cost overruns and schedule slippages are fueled in part by 
the launch of engineering and design work before hoped-for tech- 
nologies have been refined. Extraneous, often pervasive incentives, 
push program officials toward artificial deadlines and premature 
production commitments. 

Various iterations of acquisition reform at DOD have attempted 
to address these problems and reinvigorate a hidebound acquisition 
culture inside and outside the Pentagon. In launching the $200 bil- 
lion Joint Strike Fighter [JSF], aircraft acquisition, DOD promised 
the prgram would be a model of reform driven by affordability and 
the technical knowledge base, not by the disingenuous optimism 
and defense budget politics that proved so costly in the past. 

(l) 
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At the subcommittee’s request, the General Accounting Office 
[GAO], analyzed the JSF acquisition strategy to determine if the 
promise of reform is being fulfilled in practice. Their report re- 
leased today finds the Joint Strike Fighter program straying from 
commercial best practices and knowledge-driven benchmarks. As 
the date approaches to select a prime JSF contractor and begin en- 
gineering on the final system concepts, DOD appears ready to 
abandon quantitative measures of technological maturity and re- 
vert to the business as usual of concurrent technology development 
and product development. 

GAO recommends DOD focus on risk reduction efforts by matur- 
ing critical technologies prior entering the next phase of the JSF 
program, even if that means delaying contractor selection and con- 
tract awards beyond the planned March 2001 date. The program 
should be permitted to pursue the original low-risk acquisition 
strategy according to GAO, without the penalty of withdrawal of 
funding support. 

DOD disagrees, claiming critical technologies will be mature 
enough to proceed in final design and engineering next year. 

As the debate unfolds, the choice should not be between a fully 
funded Joint Strike Fighter and a commitment to acquisition re- 
form. We can have both. If the program succumbs to cold war ac- 
quisition habits, costs will skyrocket, the development cycle will 
stretch over the horizon, and the next generation fighter needed by 
the Air Force, the Navy, and the Marines might never fly. 

We welcome the testimony of all our witnesses on this important 
subject. 

[The prepared statement of Hon. Christopher Shays follows:] 
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Statement of Rep. Christopher Shays 
May 10, 2000 



The military' procurement holiday is about to end. Defense budgets being debated today 
on both sides of the Capitol reflect a bi-cameral and bi-partisan consensus on the need to 
modernize the aging planes, ships, weapons and equipment used to win the Cold War. Today, 
we discuss the need to modernize the acquisition systems the Department of Defense (DoD) will 
use to procure post-Cold War weapons systems. 


Just as the weaponry of the last century' won’t win the peace in the next, the acquisition 
practices of the past will not efficiently or affordably meet future defense needs. Fifteen year 
development cycles enshrine old technologies now rendered obsolete in fifteen months. Massive 
cost overruns and schedule slippages are fueled in pan by the launch of engineering and design 
work before hoped-for technologies have been refined. Extraneous, often perverse, incentives 
push program officials toward artificial deadlines and premature production commitments. 

Various iterations of acquisition reform at DoD have attempted to address these problems 
and reinvigorate a hidebound acquisition culture inside and outside the Pentagon. In launching 
the $200 billion Joint Strike Fighter (JSF) aircraft acquisition, DoD promised the program would 
be a model of reform, driven by affordability and the technical knowledge base, not by the 
disingenuous optimism and defense budget politics that proved so costly in the past. 

At the Subcommittee’s request, the General Accounting Office (GAO) analyzed the JSF 
acquisition strategy to determine if the promise of reform is being fulfilled in practice. Their 
report, released today, finds the Joint Strike Fighter program straying from commercial best 
practices and knowledge-driven benchmarks. As the date approaches to select a prime JSF 
contractor and begin engineering on the final system concept, DoD appears ready to abandon 
quantitative measures of technology maturity and revert to the "business as usual” of concurrent 
technology development and product development. 



4 


Statement of Rep. Christopher Shays 
May 10, 2000 


GAO recommends DoD focus on risk reduction efforts by maturing critical technologies 
prior entering the next phase of the JSF program, even if that means delaying contractor selection 
and contract awards beyond the planned March 2001 date. The program should be permitted to 
pursue the original, low-risk acquisition strategy, according to GAO, "w ithout the penalty of 
withdrawal of funding support." 

DoD disagrees, claiming critical technologies will be mature enough to proceed into final 
design and engineering early next year. 

As this debate unfolds, the choice should not be between a fully funded Joint Strike 
Fighter and a commitment to acquisition reform. We can have both. If the program succumbs 
to Cold War acquisition habits, costs will skyrocket, the development cycle will stretch over the 
horizon, and the next generation fighter needed by the Air Force, the Navy and the Marines 
might never fly. 

We welcome the testimony of all our witnesses on this important subject. 
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Mr. Shays. And if I could, I would like to welcome my colleague 
and ask if he has any comments to make. 

Mr. Terry. Thank you, no. I don’t have any opening statement. 
I’ll submit one in a few days for the record. 

I do want to compliment you on holding this hearing. When we 
look through the issues concerning the future of the military and 
the confidence that people have in our military and as we make a 
renewed commitment as a Congress to our military, we have got 
to ensure that the protocol is in place, the system is in place to en- 
sure that we use our money wisely, that we’re looking toward the 
future. And when I’m speaking at veterans organizations or just 
simply people that are interested in my home of Omaha, NE and 
home of Offutt Air Force Base, that’s what they want to know. 

That’s why this hearing is so important today. Since it is so im- 
portant, let me not continue to use the time, let’s hear from our 
witnesses. 

Mr. Shays. Thank you Mr. Terry. 

Just some housekeeping, if we could do that now. I ask unani- 
mous consent that all members of the subcommittee be permitted 
to place an opening statement in the record and that the record re- 
main open for 3 days for that purpose. Without objection, so or- 
dered. 

I ask further unanimous consent that all witnesses be permitted 
to include their written statements in the record. Without objec- 
tion, so ordered. 

Our first panel, we have three, is Mr. Louis Rodrigues, Director, 
National Security and International Affairs Division, U.S. General 
Accounting Office [GAO]. In fact, I think that’s referred to more as 
GAO than the full title. 

Now Mr. Rodrigues, if you could stand up, we’ll swear you in, as 
we swear in all our witnesses. Raise your right hand, please. 

[Witness sworn.] 

Mr. Shays. Note for the record our witness has responded in the 
affirmative. What we’re going to do is we’re going to run the clock 
for 5 minutes then we will flip it again for another 5. And then if 
you could conclude within the 10-minute period. 

STATEMENT OF LOUIS J. RODRIGUES, DIRECTOR, NATIONAL 

SECURITY AND INTERNATIONAL AFFAIRS DIVISION, U.S. 

GENERAL ACCOUNTING OFFICE 

Mr. Rodrigues. Thank you, Mr. Chairman. Mr. Chairman, Con- 
gressman Terry, I am pleased to be here today to discuss the appli- 
cation of best commercial practices to DOD weapons systems in 
general, the Joint Strike Fighter in particular. Before getting into 
details, I would like to emphasize the importance of the Joint 
Strike Fighter decision to reforming DOD’s weapons acquisition 
process. 

As you know, the Department is in the process of rewriting its 
directives governing systems acquisition, referred to as DOD 5000 
series. At the Department’s request, we have been participating in 
this effort through input to its working group. The objective of the 
rewrite is to bring about better, cheaper, faster outcomes in weap- 
ons programs. It is acquisition reform. Our contributions or inputs 
to this effort are based on our reports for the Senate Armed Serv- 
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ices Committee on using best commercial practices to improve 
weapons program outcomes. 

The DOD draft rewrite embodies critical features documented in 
our work. Two of these features are critical to the upcoming Joint 
Strike Fighter decision. First, the technology development must be 
separated from product development. That is, before entering engi- 
neering manufacturing development, we must have a match be- 
tween proven technologies and requirements. And second, metrics 
to accurately measure technology must be used. 

In the 5000 series rewrite, they are adopting a measurement sys- 
tem we used in our Joint Strike Fighter assessment, referred to as 
technology readiness levels. The commitment to this knowledge- 
based versus the current schedule- and funding-driven process is 
reflected in the testimony of the Deputy Under Secretary of De- 
fense for Acquisition Reform on March 16th before the House Gov- 
ernment Reform Subcommittee on Government Management, Infor- 
mation, and Technology, “In the new systems acquisition environ- 
ment, key acquisition and long-term funding commitments will not 
be made until technology is mature.” 

I have a lot of respect for the people in DOD who are leading ac- 
quisition reform. Philosophically we are in agreement on the best 
practices and the changes that are needed in the DOD environment 
to make such practices work on weapons systems. We at GAO are 
extremely encouraged by the commitment of DOD acquisition lead- 
ers to improving the weapons acquisition outcomes through the use 
of a knowledge-based commercial business practices. At the same 
time, however, we are concerned that the written directives and 
oral commitments will have little impact if not reflected in key de- 
cisions. 

In that sense, the key acquisition decision of entering engineer- 
ing and manufacturing development on the Joint Strike Fighter 
stands out as the flagship for weapons acquisition reform. To apply 
anything less than the standards in the directives will send a clear 
message that while the instructions and rhetoric are changing, it’s 
business as usual. 

I will now briefly describe what we have learned from our best 
practices work and how we have applied that to assessing the Joint 
Strike Fighter program. Our best practices work has shown that a 
knowledge-based process is essential to getting better cost schedule 
and performance outcomes. This means that the decisionmakers 
must have virtual certainty about critical facets of the product 
under the development when needed. This knowledge can be meas- 
ured in three junctures that we refer to as knowledge points, as 
shown in the chart to my right. 

Knowledge point 1 is when a match is made between the cus- 
tomer’s requirements and the available technology. This occurs 
prior to entering product development. 

Knowledge point 2 is when the product’s design is determined to 
be capable of meeting performance requirements. This occurs about 
midway through the product development phase. 

And knowledge point 3 is when the product is determined to be 
producible within cost, schedule, and quality targets, which occurs 
prior to entering production. 
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Today I’ll focus on only knowledge point 1 because it is the big- 
gest contributor to a successful product development, achieving 
subsequent knowledge points depends on it, and it is the point 
where JSF should be as it enters EMD. As a technology is devel- 
oped, it moves from a concept to a feasible invention to a compo- 
nent that must fit onto a product and function as expected. In be- 
tween, there are increasing levels of demonstration that can be 
measured. 

In our review of best practices for including new technology and 
products, we applied a scale of technology readiness levels from 1 
to 9, pioneered by NASA and adapted by the Air Force Research 
Laboratory. Without going into the details of each level, a level 4 
equates to a laboratory demonstration of technology that is not in 
a usable form. 

Imagine an advanced radio technology that can be demonstrated 
with components that take up a table top. When initial hand-built 
versions of all the radio’s basic elements are hand wired and tested 
together in a laboratory, the radio reaches a readiness level of 5. 

A technology readiness level of 7 is the demonstration of the 
technology that approximates its final form and occurs in an envi- 
ronment outside the laboratory. That same radio at level 7 would 
be installed and demonstrated in a platform such as an existing 
fighter aircraft. The lower the level of the technology at the time 
it is included in product development, the higher the risk that it 
will cause problems in product development. 

According to the Air Force Research Laboratory, level 7 enables 
the technology to be included on a product development with ac- 
ceptable risk. 

When we asked leading commercial firms to apply these stand- 
ards to their own methods of assessing technology maturity, we 
found that most insisted on even higher levels of readiness before 
they allowed a new technology into product development. 

Regarding the Joint Strike Fighter, in conjunction with the pro- 
gram office and the two competing contractors, we determined the 
readiness levels of critical technologies. The table to my right 
shows the technology readiness levels of eight critical technology 
areas identified by the Joint Strike Fighter program office. 

Let me try to explain this a little bit. What we had them do is 
score the technologies at three points. The blue line reflects the 
readiness level for each of those technology areas where they were 
at program launch. The yellow line reflects where they were at the 
time they did the scoring. And the red extension reflects where 
they plan to be based on what they’re planning to do between now 
and the down select to the engineering and manufacturing develop- 
ment phase. So it’s the totality of the line that shows where they 
plan to be when they enter EMD. 

In terms of engineering and manufacturing development, which 
is reflected by the second diamond on the right, none of the critical 
technology areas are projected to be at readiness level 7, which the 
Air Force Research Laboratory considers acceptable for entry into 
engineering and manufacturing development. Should any of these 
technologies be delayed, or worse, not available for incorporation 
into the final JSF design, the impact on the program would be dra- 
matic. For example, if one of the critical technologies needed to be 
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replaced with its planned backup, DOD could expect an increase of 
about 10 percent in unit costs. The backup technology would also 
significantly increase aircraft weight which can negatively impact 
aircraft performance. This technology is projected to be at a tech- 
nology readiness level of 5 at the beginning of the engineering 
manufacturing development phase, substantially below the criteria 
of 7. 

As noted earlier, at the policy level, DOD officials have agreed 
that technology development should be kept separate from product 
development and that technology readiness levels are a valid way 
to assess technology maturity. However, in response to our report 
on the Joint Strike Fighter, DOD balked at the use of technology 
readiness levels and their implications for keeping technology de- 
velopment out of the fighter’s engineering and manufacturing de- 
velopment phase. One of the reasons DOD cited for its unwilling- 
ness to accept the technology readiness levels assessed was that 
the levels were based on integration in the Joint Strike Fighter air- 
craft. 

On the contrary, the technology readiness levels assessed by the 
program office and the contractors were based on a clear under- 
standing that a level 7 could be reached by demonstrating the tech- 
nology in a relevant environment. It was further made clear that 
a relevant environment would include demonstrating a technology 
in an existing aircraft like an F-16, not a Joint Strike Fighter. 
There is no misunderstanding. 

A second reason DOD disagreed with the readiness levels as- 
sessed was that its own risk mitigation plans and judgment were 
more meaningful and that they showed the technology risk to be 
acceptable. Risk mitigation plans and judgment are necessary to 
managing any major development effort. However, without an un- 
derpinning such as technology readiness levels that allows trans- 
parency into program decisions, these methods allow significant 
technical unknowns to be judged acceptable risks because a plan 
exists for resolving them in the future. 

Experience on previous programs has shown that such methods 
have rarely assessed technical unknowns as a high or unacceptable 
risk. Consequently, they fail to guide programs to meet promised 
outcomes. Technology readiness levels are based on demonstrations 
of how well technology has actually performed. Their strength lies 
in the fact that they characterize knowledge that exists rather than 
plans to gain knowledge in the future. 

In conclusion, we believe that separating technology development 
from product development can create conditions for a successful 
Joint Strike Fighter program. To proceed as planned, entering the 
engineering manufacturing development phase of the program with 
immature technologies, is to risk scheduled delays and cost growth. 
Instead, the program has an opportunity to mature technologies in 
a more risk-tolerant environment by making the right decisions 
now. 

In our report, we recommend that the Joint Strike Fighter pro- 
gram continue in its current program definition and risk reduction 
phase, delaying the decision to move into engineering and manufac- 
turing development until technologies are demonstrated to accept- 
able levels. Taking the additional time to mature the technologies 
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will then allow the program manager to focus on design and manu- 
facturing risks during engineering and manufacturing develop- 
ment. It also increases the possibility of completing product devel- 
opment in a more timely and predictable manner. Such a delay 
does not necessarily lengthen the total product development cycle. 
In fact, the knowledge gained from time spent developing tech- 
nologies in the beginning can often shorten the time it takes to get 
the product to market. 

Similarly, a delay should not be misinterpreted as a lessening of 
support for the Joint Strike Fighter program. Rather, it would 
demonstrate decisionmakers’ willingness to make the up-front in- 
vestment necessary to mature key technologies before committing 
the Joint Strike Fighter team to deliver a product. Such a commit- 
ment is more likely to put the program on a better footing to suc- 
ceed than placing the burden on the engineering and manufactur- 
ing development phase. 

Mr. Chairman, that concludes my statement and I would be glad 
to answer any questions you or the other members may have. 

[The prepared statement of Mr. Rodrigues follows:] 
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Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to discuss the application, of best 
commercial practices to Department of Defense (DOD) weapon 
acquisitions in general and to the Joint Strike. Fighter in particular. 

After having done hundreds of reviews of major weapon acquisitions over 
the last 20 years, we have seen many of the same problems recur — cost 
increases, schedule delays, and performance problems. Over the last 4 
years, we have undertaken a body of work to examine weapon acquisition 
issues from a different, more cross-cutting perspective. Specifically, we 
have examined the best product development practices of leading 
commercial firms. Collectively, our reviews have included the practices of 
over 20 leading commercial firms that represent a variety of industries, 
including electronics, satellite communications, automotive, medical, and 
aircraft. Leading commercial firms are getting the kind of results that DOD 
seeks: more sophisticated products developed in less time and cost than, 
their predecessors. Our work shows that DOD can learn valuable lessons 
from the commercial sector to get better and more predictable outcomes 
from weapon system development programs. A listing of the reports we 
have issued on best practices that can be applied to weapon acquisitions is 
included in the appendix. 

DOD has taken steps to reflect best commercial practices in its acquisition 
policies. However, the real test of these policies is in how they influence 
individual decisions, such as the upcoming engineering and manufacturing 
development decision on the Joint Strike Fighter program. This program is 
to produce three fighter variants to meet multiservice requirements: 
conventional flight for the Air Force, short take-off and landing for the 
Marine Corps, and carrier operations for the Navy. The program will also 
provide aircraft to the British royal Navy and Air Force. As currently 
planned, the program will cost about $200 billion to develop and procure 
over 3,000 aircraft and related support equipment 

My testimony focuses on the best commercial practices for developing 
new products, the reasons why DOD does not follow these practices, and 
the opportunity that Joint Strike Fighter represents to strengthen — or 
weaken — the effect of best practices and acquisition reform on major 
weapons. 


Results in Brief Leading commercial firms have adopted a knowledge-based approach to 

developing new products, underwritten by incentives that, encourage 
realism, candor, and meeting product expectations. Making sure that new 


Page 1 


GAQ/T-NSIAD-00-.1 73 



12 


technology is mature — that is, demonstrating that it works — before a 
product development starts is the foundation for this approach. Leading 
commercial firms discipline the product development phase by adhering 
to (1) time limits for completing development and (2) high standards for 
demonstrating design and production knowledge. These practices have 
put commercial managers in an excellent position to succeed in 
developing better products in less time and producing them within 
estimated costs. To do otherwise would risk failure in the form of 
customer dissatisfaction. 

DOD programs, with some exceptions, proceed with lower levels of 
knowledge about key factors of product development and allow 
technology development, to take place during product development. DOD’s 
variances from best practices stem from strong incentives for starting 
programs too early; overpromising performance capabilities; and 
understating expected costs, schedules, and technical risks. While these 
incentives evolved over time to help build support for programs, they put 
program managers in a very difficult position to deliver better weapons on 
time and within budget. Moreover, there is little risk that the DOD 
customer will be dismayed by a program not being delivered as promised. 
DOD accepts the need to get better outcomes from its weapon system 
programs and accepts best commercial practices as away to get those 
outcomes. In fact, it is currently incorporating such practices in a major 
revision of its acquisition policy. However, new policies will not produce 
better program outcomes unless they influence the decisions made on 
individual weapon systems. 

DOD has designated the Joint. Strike Fighter as a flagship program for 
acquisition reform. Funding requests are now' before the Congress to 
support, the Joint Strike Fighter's April 2001 entry 7 into engineering and 
manufacturing development By best practices standards, none of the 
fighter’s critical technology areas identified by the program office are 
expected to be at readiness levels considered an acceptable risk for entry 
into engineering and manufacturing development. Delaying this phase of 
the program until these technologies are mature would improve the 
chances that the Joint Strike Fighter will be fielded as planned. However, 
despite not having the requisite knowledge for the eight technologies, 

DOD has deemed the risks manageable and proposes to proceed with the 
program as planned. Such a decision reinforces traditional incentives and 
increases the likelihood for future cost, schedule, and performance 
problems- DOD’s plans to move the Joint Strike Fighter into engineering 
and manufacturing development with immature technology illustrates a 
lack of commitment to following best commercial practices as part of its 
acquisition reform efforts. 
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A Best Practices 
Model for Acquisition 


The characteristics of best commercial practices suggest a process for 
developing new capabilities — whether they are commercial or defense 
products — that is based on knowledge. It is a process in which technology 
development and product development are treated differently and 
managed separately. The process of developing technology culminates in 
discovery and must, by its nature, allow room for unexpected results and 
delays. The process of developing a product culminates in delivery, and 
therefore, gives great weight to design and production. Discipline is 
inherent because criteria exist, tools are used, and a program does not go 
forward unless a strong business case on which the program was 
originally justified continues to hold true. 


We have learned that a knowledge-based process is essential to getting 
better cost, schedule, and performance outcomes. This means that 
decision makers must have virtual certainty about critical facets of the 
product under development when needed. Such knowledge is the inverse 
of risk. Commercial and military programs do not all follow the same 
processes in their development cycles. However, at some point, full 
knowledge is attained about a completed product, regardless of the 
development approach taken. This knowledge can be measured at three 
key junctures that we refer to as knowledge points: 

• Knowledge point 1: when a match is made between the customer’s 
requirements and the available technology; 

• Knowledge point 2: when the product’s design is determined to be capable 
of meeting performance requirements; and 

• Knowledge point 3: when the product is determined to be producible 
within cost, schedule, and quality targets. 


We have identified metrics that indicate these knowledge levels and can 
thus help forecast outcomes as a development program progresses. A best 
practices model for technology development and product development is 
depicted in figure 1. 
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Figure 1 : Best Practices lor Developing Technology and Products 
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Commercial firms gain more knowledge about a product’s technology, 
design, and producibility much earlier than DOD acquisition programs we 
reviewed. Two features of leading commercial products stand out for 
making a manageable product development. First, there is a clear break 
between technology development and product development. The launch of 
a new product development, in commercial ventures is a clearly defined 
undertaking and before beginning, firms insist on having the technology in 
hand that is needed to meet customer requirements. Second, leading firms 
limit the length of time it takes to develop the product. This limit is key to 
getting the product to market and focuses attention on the design and 
production knowledge points. It also provides discipline to the technology 
development process to ensure product development will not be launched 
until the technology match is made. 

The leading commercial Arms we have visited consciously limited their 
product developments from 18 months to just over 4 years. They 
understand that this keeps product development within a time frame that 
can keep people focused on delivering a product. In fact, one commercial 
executive observed that it is unreasonable to expect people to focus on a 
goal like production start that is more than 4 years away. The limited time 
frames provide strong incentives for a commercial manager to keep 
immature technologies out of the product design, In fact, these time 
frames give product managers clout in fostering cost and performance 
trade-offs before the program begins, ultimately limiting a product’s 
requirements to what can be achieved with demonstrated technologies 
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within the specified time period. To live within these time constraints and 
keep innovation alive, commercial firms have adopted an evolutionary 
approach; they saw requirements that cannot be met with proven 
technologies for the next iteration of the product. Commercial firms also 
found that limiting product development time frames 

makes it easier to hold people accountable for meeting promised cost, 
schedule, and performance targets; 

enables a production-oriented focus throughout product development, 
providing incentives for identifying risks early; 

makes product development costs and schedule more predictable; and 
allows firms to get into production and, therefore, to the point of sale 
quicker. 

Once a product development is under way, the firms demand — and 
receive — specific knowledge about the design and producibility of the 
product before production begins. The process of discovery — the 
accumulation of knowledge and elimination of unknowns — is completed 
well ahead of production. There is a synergy in this process, as the 
attainment of each successive knowledge point builds on the preceding 
one. 

In contrast, DOD programs are started earlier and allows technology 
development to continue into product development and even into 
production. Consequently, the programs proceed with much less 
knowledge available — and thus more risk— about required technologies, 
design capability, and producibility. This approach to technology and 
product development is shown in figure 2. 
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Figure 2: Levels of Knowledge Attained in DOD Weapon System 
Developments 
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Proceeding with lower lex-els of knowledge available means that, during 
product development, maturity of technology, the design, and production 
methods must ail be pursued at the same time. The rippling effect of 
discovering and overcoming problems in product development explains 
much of the turbulence in DOD program outcomes. Metrics, such as 
technology readiness lex r els and percent of engineering drawings 
complete, can be used to predict these consequences. Product 
development times, long to begin with, stretch even further in reaction to 
problems. We calculate that they can take 3 to 10 times as long as 
commercial products. 


Knowledge Point 1: 
Requirements and 
Technology Are Matched 


Technology development has the ultimate objective of bringing a 
technology up to the point that it can be readily integrated into a new 
product and counted on to meet requirements. We have found that getting 
the match between customer requirements and mature technology to be 
the biggest contributor to a successful product development. 3 As a 


3 For more information, see Best Practices: Better Management oCTechnoIogy Development Can 
Improve Weapon System Outcomes (GAO/NS1&D-99-162, July 30, 1999). 
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technology is developed, it moves from a concept to a feasible invention to 
a component that must fit onto a product and function as expected. In 
between, there are increasing levels of demonstration that can be 
measured. In our review of best practices for including new technology in 
products, we applied a scale of technology readiness levels — from one to 
nine— pioneered by the National Aeronautics and Space Administration 
and adapted by the Air Force Research Laboratories. 

Without going into the details of each level, a level four equates to a 
laboratory demonstration of a technology that is still not in a usable form. 
Imagine an advanced radio technology that can be demonstrated with 
components that take up a table top. When initial hand-built versions of 
all of the radio’s basic elements are hand-wired and tested together in a 
laboratory, the radio reaches a readiness level of five. A technology 
readiness level of sever, is the demonstration of a technology that 
approximates its final form and occurs in an environment outside the 
laboratory. The same radio at level seven would be installed and 
demonstrated in a platform, such as a fighter aircraft. 

The lower the level of the technology at the time it is included in a product 
development program, the higher the risk that it will cause problems in the 
product development According to the Air Force Research Laboratory, 
level seven enables a technology to be included on a product development, 
with acceptable risk. When we asked leading commercial firms to apply 
these standards to their own methods for assessing technology maturity, 
we found that most insisted on even higher levels of readiness before they 
allowed a new technology into product development. When we examined 
weapon systems that experienced cost and schedule problems, we found 
that they started with key technologies at levels three and four. By the 
time the programs reached a point DOD considers analogous to beginning 
product development, key technologies were still at. level five or lower. 
Conversely, DOD programs for which key technologies were at 
significantly higher levels of maturity at the start, had not experienced 
such problems. 


Knowledge Point 2: The 
Design Will Perform as 
Required 


By the halfway point in product development, leading commercial firms 
achieve near certainty that their product designs would meet customer 
requirements and have gone a long way toward ensuring that the products 
can be produced . 2 Both DOD and the commercial firms hold a critical 


- For more information, see Best Practices; Successful Application to Weapon Acquisitions Require 
Changes in DOD's Environment (GAO/NSLA.D-98-56, Feb. 24. 1998). 
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design review to review engineering drawings, confirm the design is 
mature, and “freeze” the design to minimize changes in the future. The 
completion of engineering drawings and their release to manufacturing 
organizations signify that program managers are confident in their 
knowledge that the design performs acceptably and is mature. The 
drawings are critical to documenting this knowledge because they are 
precision schematics of die entire product and all of its component parts. 
They also reflect the results of testing and simulation and describe the 
materials and manufacturing processes to be used to make each 
component. 

Both DOD and commercial companies consider the design to be 
essentially complete when about 90 percent of the engineering drawings 
are completed. Officials from commercial companies such as Boeing and 
Hughes told us that they typically had over . 90 percent of these drawings 
available for the critical design review. The'DOD programs we reviewed 
had less than 60 percent — one had less than one-third — of the drawings 
done at the time their critical design reviews were held. Thus, these 
programs had significantly less knowledge about their designs. The 
programs did not get or were not expected to get to the 90-percent level of 
completion on the drawings until late in development or in production. 
Nonetheless, at the time of the critical design reviews, the risks of 
proceeding with the rest of development on these programs as planned 
were deemed acceptable. The programs however encountered significant 
design problems in testing that occurred after the critical design review. 


Knowledge Point 3: 
Production Units Will Meet 
Cost, Quality, and 
Schedule Objectives 


Leading commercial firms reach the point at which they know that 
manufacturing processes will produce a new product conforming to cost, 
quality, and schedule targets by the end of product development — before 
they begin fabricating production articles. Reaching this point means more 
than knowing the product could be manufactured; it means that all key 
processes are under control, such that the quality, volume, and cost of 
their output are proven acceptable. Commercial firms relied on good 
supplier relationships, known manufacturing processes, and statistical 
process control to achieve this knowledge early and. in fact, had all then- 
key processes under statistical process control when production begins. 
All of the companies we visited agreed that a high level of knowledge 
about technology and design early in the process makes the control of 
processes possible. 


DOD programs did not have nearly this level of knowledge at production. 
One weapon system program that had been in production for nearly 9 
years at the time of our 1998 review still had less than 13 percent of its key 
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manufacturing processes in control. Another program had 40 percent of its 
key manufacturing processes in control 2 years before production was 
scheduled to begin but was not scheduled to have all key processes in 
control until 4 years into production. Both programs experienced basic 
producibility problems that were not discovered until late in development 
or early in production. These risks went unrecognized even though DOD 
had established criteria for determining whether risks were acceptable 
and whether enough knowledge had been gained to enter the next 
development phase. 


Changes in DOD 
Environment Needed 
to Adopt Best 
Practices 


The most important factors in the adoption of best practices are the 
incentives perceived by managers of technology and product 
developments. The differences in the practices employed by the leading 
commercial firms and DOD reflect the different demands imposed on 
programs in their environments. The way success and failure are defined 
for commercial and defense product developments differs considerably 
and results in different incentives, evoking different behaviors from the 
people managing the programs. Specific practices take root and are 
sustained because they help a program succeed in its environment. 


Incentives in the Leading commercial firms begin product development only when a solid 

Pnmmorrinl Fnvirrvnmpnt business case can be made. The business case centers on the ability to 
produce a product that the customer will buy and that will provide an 
acceptable return on investment. The point of sale occurs in production 
after development is complete; program success is determined when the 
customer buys the finished product. If the firm has not made a sound 
business case, or has been unable to deliver on one or more of the 
business case factors, it faces a very real prospect of failure — the 
customer may walk away. Also, if one product development takes more 
time and money to complete than expected, it denies the firm 
opportunities to invest those resources in other products. Because the 
match between technologies and product requirements is made before the 
product development is launched, the cost and schedule consequences 
associated with discovery are minimized. 

Production is a dominant concern throughout the product development 
process and forces discipline and trade-offs in the design process, This 
environment encourages realistic assessments of risks and costs; doing 
otherwise would threaten the business case and invite failure. For the 
same reasons, the environment places a high value on knowledge for 
making decisions. Program managers have good reasons to identify risks 
early, be intolerant of unknowns, and not rely on testing late in the process 
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as the main vehicle for discovering the performance characteristics of the 
product. By protecting the business case as the key to success, program 
managers in leading commercial firms are conservative in their estimates 
and aggressive in risk reduction. Ultimately, preserving the business case 
strengthens the ability of managers to say “no” to pressures to accept high 
risks or unknowns. Practices such as maturing technologies to high 
readiness levels before inclusion in a program, having 90 percent of 
engineering drawings done by the critical design review, and achieving 
statistical process control before production are adopted because they 
help ensure success. 


Incentives in the DOD The basic management goal for a weapon system program in DOD is 

Environment similar to develop and deliver a product that meets the customer’s needs. 

However, the pressures of successfully competing for the funds to start 
and sustain a DOD program provide different incentives. Compared with 
commercial programs, the DOD environment encourages launching 
product developments that embody more technical unknowns and less 
knowledge about the performance and production risks they entail. A new 
weapon system can be more readily defended if it possesses performance 
features that significantly distinguish it from other systems. Consequently, 
aspiring DOD programs have incentives to include performance features 
and design characteristics that rely on immature technologies. These 
unknowns place a much greater focus on maturing technology during 
product development than we found on commercial programs. 

Even though less infoimation about a new product development is 
available at the time DOD programs are launched, the competition for 
funding forces detailed projections to be made from that information. A 
product development cannot be launched unless the program’s 
development and production cost, as well as timing, falls within available 
funding. Because DOD relies largely on forecasts of cost, schedule, and 
performance that are comparatively soft at the time, success in competing 
for funding encourages managers to squeeze cost and schedule estimates 
into profiles of available funding. Additional requirements, such as high 
reliability and maintainability, serve to make the fit even tighter. As 
competition for funding wall continue throughout the program’s 
development, success is measured in terms of ability to secure the next 
installment. 

The risks associated with developing new technologies together with the 
product — within tight estimates— are deemed acceptable. Production 
realities, critical to matching technological capabilities with customer 
requirements on commercial programs, are too far away from the DOD 


GAO/T-NSIAD-OO-l 73 



21 


launch decision to have the same curbing effect on technology decisions. 
The environment for managing weapon system programs is particularly 
difficult for managing technology development. The ups and downs and 
resource changes associated with the technology discovery process do not 
mesh well with a program’s need to meet cost, schedule, and performance 
goals. Problems with developing technologies, winch are to be expected, 
can actually threaten the support for a program if they become known. 

These pressures and incentives explain why the behavior of weapon 
system managers differs from managers of commercial product 
developments. Problems or indications that the estimates are being 
breached do not help sustain funding support for the program in later 
years, and thus their admission is implicitly discouraged. An optimistic 
cost estimate makes it easier to launch a product development and sustain 
annual approval; admission that costs are likely to be higher could invite 
failure. Rewards for discovering and recognizing potential problems early 
in a DOD product development are few. Less available knowledge makes it 
harder for program managers to say “no.” In contrast with leading 
commercial firms, not having attained knowledge — on the full 
performance of a key technology or the true risks facing manufacture, for 
example, — can be perceived as better than knowing that problems exist. 
For these reasons, the practices associated with managing to knowledge 
standards — such as for technology, design, arid production maturity — are 
not readily adopted in DOD. 

These observations about the differences between the commercial and 
DOD environments should not be interpreted to mean that commercial 
managers are somehow more skilled or knowledgeable than their DOD 
counterparts. DOD program managers act in response to the pressures 
they face. All of the numerous participants in the acquisition process play 
a part in creating these pressures. Commercial program managers are put 
in a better position to succeed; they have to worry only about product 
design and production within the cost, schedule, and performance 
demands of the business case. 


Charting a Course for 
Adopting Best Practices to 
Get Better Outcomes 


Commercial practices for gaining knowledge and assessing risks can help 
produce better outcomes on weapon systems. Collectively, better 
individual outcomes will help DOD to attain modernization goals and 
improve funding stability for programs. For such practices to w’ork, 
however, the knowledge they produce must help a DOD program succeed 
in its environment. Thus, the DOD environment must become conducive to 
such practices. At least two factors are critical to fostering such an 
environment. First, managers must be relieved of the need to overpromise 
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on performance and resource estimates at the program launch decision. 
Separating technology development so that it does not have to be managed 
within the confines of a weapon system program would go a long way to 
relieving this pressure. Clearly, DOD has to develop technology, 
particularly the technology that is unique to military applications. 

However, by separating technology development from weapon programs, 
DOD could insist on higher standards for knowledge on its programs and 
get better results when those programs transition to production. 

Second, once a program is under way, the participants in the acquisition 
process must make it acceptable for managers to identify unknowns as 
high risks so that they can be aggressively worked on earlier in 
development If the Congress and DOD weighed program launch decisions 
and subsequent progress on weapon systems by applying a common set of 
knowledge standards, like those gleaned from leading commercial firms, 
they could create a better business case for starting a weapon system 
program. By developing technology separately to high readiness levels 
before including it in a program and by adhering to standards such as 
knowledge points in product development, DOD program managers can be 
put in a better position to succeed in the timely design and production of 
weapon systems. The shorter cycle times associated with these practices 
could make it possible to better align the tenures of program managers 
with the product development phase, making them more accountable for 
program outcomes. 

The Office of the Under Secretary of Defense (Acquisition, Technology, 
and Logistics) supports shorter weapon system development times and 
more aggressive pursuit of technology outside of programs. It also 
supports the use of best commercial practices, such as taking an 
evolutionary approach to developing new products. DOD is capturing 
these and other practices in a substantial revision of the regulations that 
guide the management of weapon system programs. These regulations are 
currently in draft form. 

The real test of the participants' resolve to get better outcomes by applying 
best practices will be the decisions made on individual weapon systems, 
such as for launch and funding. These decisions define what success 
means in DOD and what practices contribute to success. Decisions made 
by DOD or the Congress to advance or fund programs that do not have 
enough knowledge to meet agreed-upon standards signal to managers that 
not having the necessary level of knowledge is acceptable. On the other 
hand, participants should support decisions to not start new programs that 
need technology advances to meet unforgiving requirements or to 
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recognize early that a change in a program is necessary to attain desired 
knowledge levels. 


The Decision to Move 
the Joint Strike 
Fighter Into 
Engineering and 
Manufacturing 
Development: A Case 
in Point 


The Joint Strike Fighter program, and its implications for best practices 
and true acquisition reform, is a good test of whether the desire for better 
outcomes can outweigh traditional pressures to get a program approved. 
In our recent review of the Joint Strike Fighter program, we employed 
knowledge standards consistent with best practices and DOD acquisition 
reforms. 3 DOD has designated the Joint Strike Fighter program as a 
flagship program for acquisition reform. The Joint Strike Fighter program, 
the most expensive aircraft program in DOD, is at a critical juncture in its 
acquisition cycle. It is approaching the point where DOD must decide 
whether to commit to the engineering and manufacturing development 
phase — analogous to a commercial product launch decision. During 
engineering and manufacturing development, the Joint Strike Fighter will 
be fully developed, engineered, designed, fabricated, tested, and evaluated 
to demonstrate that the production aircraft will meet stated requirements. 
This phase is estimated to cost $20 billion, require annual funding levels as 
high as $4 billion, and last about 8 years. 


The best practice for such a decision is to have a match between mature 
technologies and weapon requirements. It represents the first knowledge 
point. The Joint Strike Fighter does not meet this standard; several 
technologies that are critical to meeting requirements will not be 
sufficiently mature. Consequently, the Joint Strike Fighter will not enter 
the engineering and manufacturing development phase with low technical 
risk. However, DOD would like to go forward with the program anyway. 
Doing so would have two major consequences. First, it would put the 
program on a path that has yielded cost growth and schedule slippage on 
many previous programs. Second, as Joint Strike Fighter is the largest 
acquisition in the foreseeable future, it will send signals to other programs 
that best practices and acquisition reform need not be heeded when it 
comes to major weapon systems. 


Joint Strike Fighter 
Requirements Depend on 
Immature Technologies 


While we have been encouraged by the design of the Joint Strike Fighter 
acquisition strategy, we have become concerned about its implementation. 
Our biggest concern is that critical technologies for meeting affordability 
and performance requirements are projected to be at low levels of 


3 Joint Strike Fighter Acquisition: Development Schedule Should Be Changed to Reduce Risks 
(GAO/NSLAD-OO-74, May 9. 2000). 
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maturity if the engineering and manufacturing development contract 
begins as scheduled in April 2001. In other words, the program may not 
achieve a fundamental element of a knowledge-based process — the 
separation of technology development from product development — as it 
begins full-fledged product development activities. This means that the 
Joint Strike Fighter program manager will be responsible for developing 
technologies while concurrently designing, building, and testing the 
prototype Joint Strike Fighter aircraft. On past programs, such 
concurrency has caused significant cost growth and schedule delays. This 
is the first, and perhaps most important, knowledge point exhibited in best 
practices, because lack of knowledge about technologies means the 
program manager cannot fully focus on design and manufacturing issues. 
This additional risk makes product development cycle time and cost less 
predictable. In addition, once in a product development environment, 
external pressures to keep a program moving become dominant, such as 
preserving cost and schedule estimates to secure budget approval. 

In our recently issued report, we evaluated the maturity of key 
technologies on the Joint Strike Fighter program. At our request, the 
program office identified eight technology areas that are considered 
critical to meeting the fighter’s cost, schedule, and performance 
objectives. In conjunction with the program office and the two competing 
contractors, we determined the readiness levels of these technologies 
needed to meet Joint Strike Fighter performance requirements at three 
points in time: when the Joint Strike Fighter program was started in 1996, 
when we conducted our review in December 1999, and when the program 
is scheduled to enter engineering and manufacturing development. Those 
assessments showed that when the Joint Strike Fighter program was 
started, most of the critical technologies were well below the readiness 
levels considered acceptable risk to begin a program. The technology 
readiness levels of the eight critical Joint Strike Fighter technology areas 
are shown in figure 3. 
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Figure 3: Joint Strike Fighter Critical Technology Readiness Levels at 
Program Start and as Projected for Entry into Engineering and 
Manufacturing Development 
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As figure 3 shows, all of the critical technology areas are expected to be at 
readiness levels lower than the level seven considered acceptable risk for 
entry into engineering and manufacturing development. Six of the 
technologies will still be below the readiness level that is considered 
acceptable risk for program start, which occurred over 3 years ago for the 
Joint Strike Fighter program. Many of these will only be demonstrated in 
laboratories or in ground tests when the engineering and manufacturing 
development phase starts. They have a considerable amount of 
development remaining before they are considered mature. Moreover, as a 
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result of cost growth and schedule concerns, in May 1999, DOD delayed 
some technology demonstrations until after the engineering and 
manufacturing development phase begins. 

Should any of these technologies be delayed or, worse, not available for 
incorporation into the final Joint Strike Fighter design, the impact on the 
program would be dramatic. For example, if one of the above critical 
technologies needed to be replaced with its planned backup, DOD could 
expect an increase of several billion dollars in production and operation 
and support costs. 4 The backup technology would also significantly 
increase aircraft weight, which can negatively impact aircraft 
performance. This technology is expected to be at unacceptable readiness 
levels at the beginning of the engineering and manufacturing development 
phase, which indicates that substantial technology development must still 
occur during this phase. 

As noted above, at the policy level, DOD officials have agreed that 
technology development should be kept separate from product 
development and that technology readiness levels are a valid way to assess 
technology maturity. However, in its response to our report on the Joint 
Strike Fighter — an individual program decision — DOD balked at the use of 
technology readiness levels and their implications for keeping technology 
development out of the Joint Strike Fighter’s engineering and 
manufacturing development phase. One of the reasons DOD cited for its 
unwillingness to accept the technology readiness levels assessed for the 
eight Joint Strike Fighter technologies was that the levels were based on 
integration in the Joint Strike Fighter aircraft — too high a standard. On the 
contrary, the technology readiness levels assessed by the program office 
and the contractors were based on a clear understanding that a level seven 
could be reached by demonstrating a technology in a relevant 
environment. It was further made clear that a relevant environment would 
include demonstrating a technology in an existing aircraft like an F-16, not 
a Joint Strike Fighter. 

A second reason DOD disagreed with the readiness levels assessed for the 
eight technologies was that its own risk mitigation plans and judgment 
were more meaningful and that they showed the technology risk to be 
acceptable. Risk mitigation plans and judgment are necessary to managing 
any major development effort like the Joint Strike Fighter. However, 


4 Specific details cannot be provided due to the competitive nature of the Joint Strike Fighter program. 


Page 16 


GAO/T-NSlAD-OO-173 



27 


without an underpinning such as technology readiness levels that allows 
transparency into program decisions, these methods allow significant 
technical unknowns to be judged acceptable risks because a plan exists 
for resolving the unknowns in the future. Experience on previous 
programs has shown that such methods have rarely assessed technical 
unknowns as a high or unacceptable risk; consequently, they failed to 
guide programs to meet promised outcomes. Technology readiness lewis 
are based on actual demonstrations of how well technologies actually 
perform. Their strength lies ‘in the fact that they characterize knowledge 
that exists rather than plans to gain knowledge in the future; they are thus 
less susceptible to optimism. A clear picture of knowledge — or its 
absence — may be more likely to prompt action than a favorable risk 
assessment. 


An 8-Year Phase That 
Spans Technology 
development to 
Production Represents a 
Herculean Management 
Undertaking 


The Joint Strike Fighter program began in 1996 and will not deliver its first, 
operational aircraft until 2008. With the 4 years spent in concept 
demonstration and the 8 years to be spent in engineering and 
manufacturing development, the result is a 12-year development cycle. 
This is much longer than the development cycles of leading commercial 
firms and double the goal set by the Office of the Secretary of Defense. 
Much of this long development cycle is the result of the need to mature 
technologies that have not yet been demonstrated as ready to meet key 
cost and performance requirements. DOD has highlighted the Joint Strike 
Fighter program as one that will make significant cost/performance trade- 
offs in order to develop an affordable aircraft. However, DOD’s desire to 
have a low cost aircraft that must also meet demanding requirements has 
limited the range of technological solutions and has necessitated the 
pursuit for new technologies. 

Traditionally, a weapons final performance requirements are developed 
early in a program, or in many cases before the program begins. In the 
case of Joint Strike Fighter program, requirements were finalized much 
later in the acquisition cycle. Program officials stated that this provided 
the program flexibility to conduct cost and performance trade-offs before 
requirement and design decisions became final. While this approach is 
consistent with best, practices, it has not adequately taken into account the 
readiness of the critical technologies. Many of the trade-offs that were 
made involved decisions to bring technologies that were not yet. 
demonstrated into the engineering and manufacturing phase of the 
program. Thus, the program does not have a baseline design based on 
demonstrated technologies that could be developed in cycle times 
commensurate with best practices. 
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As the Joint Strike Fighter program approaches the engineering and 
manufacturing development decision point, it is not in a good position to 
succeed, if success means delivering the aircraft on time and within 
budget The design requires significant technological invention in order to 
satisfy all the user’s requirements. According to Joint Strike Fighter 
officials, an objective of the engineering and manufacturing development 
phase is bringing the technologies up to maturity levels that will allow 
them to be incorporated onto a Joint Strike Fighter. Therefore, at a time 
when the program should be focused on designing and building the 
air craft, the Joint Strike Fighter pr ogram will have to put significant effort 
and resources into demonstrating that key technologies are ready for 
inclusion onto the product. As a result, the progrant has planned almost 8 
years for its engineering and manufacturing development phase. The 
length and scope of the effort operate against the ability to estimate cost 
and completion schedules. 

Commercial firms have established practices to limit product development, 
cycle times, thereby increasing the possibility that program managers will 
remain on programs until they are complete. Holding one program 
manager accountable for the content of a product at the time the launch 
decision is made encourages that person to raise issues and problems 
early and not oveipromise the capabilities of a new product by relying on 
immature technologies. This puts the manager in a good position to deliver 
a high quality product, on time, and on budget 

Since the next phase of the Joint Strike Fighter program is estimated to 
last about 8 years, program managers currently involved in key decisions 
about the development plan will likely not be responsible for its 
implementation. It has already had three program managers since its 
beginning aboui 3 'A years ago. As a result, conditions to be accepted at 
engineering and manufacturing development, such as the acceptance of 
low technology readiness levels, will more than likely become the 
responsibility of another program manager. 


C Oil elusion At this P oilrt m its development, there are a number of ways for DOD to 

make the Joint Strike Fighter program’s environment more conducive to 
better cost and schedule outcomes. We believe that separating technology 
development from product development can still create conditions for a 
successful Joint Strike Fighter program. To proceed as planned — entering 
a phase of the program with immature technologies that should be focused 
on design and production — is to risk continued delays and cost growth. 
Instead, the program has an opportunity to mature technologies in a more 
risk-tolerant environment by making the right decisions now. 
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In our report, we recommend that the Joint Strike Fighter program 
continue in its current program definition and risk reduction phase, 
delaying the decision to move into engineering and manufacturing until 
technologies are demonstrated to acceptable levels. Taking the additional 
time to mature the technologies will then allow the program manager to 
focus on design and manufacturing risks during engineering and 
manufacturing development. It also increases the possibility of completing 
product development in a more timely and predictable manner. Such a 
delay does not necessarily lengthen the total product development cycle. 
In fact, the knowledge gained from time spent developing technologies in 
die beginning can often shorten the time it takes to get the product to 
market. 

Similarly, a delay should not be misinterpreted as a lessening of support 
for the Joint Strike Fighter program. Rather, it would demonstrate 
decisionmakers’ willingness to make the up-front investment necessary to 
mature key technologies before committing the Joint Strike Fighter team 
to deliver a product. Such a commitment is more likely to put the program 
on a better footing to succeed than placing the burden on the engineering 
and manufacturing development phase. 


Mr. Chairman, that concludes my statement. I will be happy to respond to 
any questions you or other Members of the Subcommittee may have. 
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Mr. Shays. Thank the gentleman very much. Thank you very 
much, Mr. Rodrigues. The bottom line is we’re scheduled to build 
the Air Force F-22 and the F/A-18, the F Super Hornet, and then 
we’re scheduled in the future to see this project go forward. You’re 
not suggesting in any way that we end this program and not do 
it; correct? 

Mr. Rodrigues. Correct, Mr. Chairman. 

Mr. Shays. The bottom line to the debate we’re going to have 
today is you just want them to follow their game plan. 

Mr. Rodrigues. We want them to 

Mr. Shays. By “they,” the DOD to follow the game plan. 

Mr. Rodrigues. Yes, sir. Mr. Chairman, what we’re talking 
about is that the Joint Strike Fighter decision is such a major deci- 
sion it is the signal of what happens in acquisition reform. It will 
underscore the implementation of the changes that are being made 
today in the regulations to guide this process. And if we make the 
same types of decisions that we have made in the past, that is, to 
allow unknowns to creep into product development, to allow tech- 
nology development to be done concurrently with product develop- 
ment, we’ll continue to see very long development cycles like we 
have seen on the F-22, like we saw on the B-2, and like we have 
seen on programs historically. 

And that situation undermines our modernization effort. It leads 
to cost growth and schedule delays. It leads to the problems that 
we have seen with the programs in the past where we end up cut- 
ting the quantities in half and never do meet our modernization 
goals. In fact, it exacerbates the problems that we’re trying to re- 
solve, one of which is aging of the fleet. It is a real issue. And to 
continue to move into programs where we can’t deliver on cost and 
schedule, that are fit into very fight funding wedges, puts us in a 
position where we end up in the future cutting the quantities be- 
cause of cost problems and ending up without the modernization 
that we absolutely need. 

Mr. Shays. The bottom line, I just want to establish the point 
that GAO is not saying that we scrap this program. You want us 
to follow the game plan. And so what this committee has to under- 
stand is what you’re saying and what DOD is saying and where 
those disagreements occur. 

I’m going to expose my ignorance a little bit here. If we could go 
to the first chart, I find that’s the best way to learn. I sometimes 
make my staff nervous when I do this. 

Mr. Rodrigues. What this depicts, Mr. Chairman is, in effect, 
the commercial model. 

Mr. Shays. What I need to do is just incorporate this with your 
nine product requirements. How does that — this isn’t the first three 
of the nine. Can you incorporate this to the — to TRLs? 

Mr. Rodrigues. TRLs are a metric. They are a metric. They’re 
the metric that allows you to make the determination of the tech- 
nology match to requirements. So that the first knowledge point in 
any program is assuring that you have a match between the tech- 
nology and the requirement. 

Mr. Shays. Is point 3 here point 7 on TRLs? 

Mr. Rodrigues. No. The 7 on the TRL is that that match has 
occurred in that technology-to-requirements match. It is the first 
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knowledge point. It is where we are approaching on the Joint 
Strike Fighter. 

Mr. Shays. And the debate, it seems to me, is whether DOD 
thinks your match point at 7 is the key point. They’re willing to 
move forward before they’ve reached 7. They dispute that they 
haven’t reached point 7. 

Mr. Rodrigues. I’m sure that will become a confusing issue. 

Mr. Shays. I just want to know what you think. 

Mr. Rodrigues. To me it’s absolutely clear. Applying technology 
readiness levels, which is the demonstration of these technologies, 
puts those technologies where they are on the second chart. 

Would you put that chart back up, please? 

And we’re accepting their projections of where they’re going to 
be. Once again as I said, the diamond to the right is where you 
should have to be. That is the level 7. That is the acceptable risk 
for entry into engineering and manufacturing development. 

Mr. Shays. What I’m asking you is do they dispute what you 
have up here and are they — believe that the acceptable risk is at 
point 6 and you believe the acceptable risk is at point 7? Where is 
the dispute as far as you would articulate it? 

Mr. Rodrigues. They argue that there was a misunderstanding 
in the application by the contractors. We did not score these. The 
contractors applied the criteria. The DOD argument, as I under- 
stand it, is that there was a misunderstanding in the application. 
To get to a level 7 requires the demonstration of the technology or 
the hardware involved in this technology in the form fit and func- 
tion. In other words, what it’s going to really look like when it has 
to go onto the Joint Strike Fighter in a relevant environment. 

And the Department’s position, as I understand it, is that there 
was a misunderstanding and that what people were thinking when 
they scored this was that we were saying it had to actually be on 
the Joint Strike Fighter. Well, that obviously cannot happen before 
the program enters engineering and manufacturing development 
because there is no Joint Strike Fighter. 

We worked very closely with the contractors and program office 
people who were at those meetings to explain exactly how to do 
this. I have the sheets that were provided to them. We spent days. 
And we made sure that they understood that we are talking about 
surrogates. There is physically no way to do the kind of demonstra- 
tion they’re contending these people were thinking we were talking 
about absent a Joint Strike Fighter. What we were talking about 
was demonstrated on surrogates. 

Mr. Shays. To give some life to these technologies, you’ve listed 
under numbers because of proprietary issues, but — so I won’t de- 
bate which is which. But what we’re talking about is propulsion, 
we’re talking about flight systems, we’re talking about weapons. 
Each of these is a technology. We’re talking about structures and 
materials. In avionics we’re talking about radar and the mission 
systems and supportability and training and producibility. Those 
are the things that we’re talking about. 

Now when we started out, they were at level technology 1, 
whichever one it is, was at level 3. And you’re then saying that 
they have gotten to level 5 and expect to get in technology 1 to 
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level 6. And then make a commitment. They’re wanting to commit 
at level 6, correct? 

Mr. Rodrigues. That’s what we’re saying. They’re not going to 
agree with that. 

Mr. Shays. But that’s what you think they’re saying. 

Mr. Rodrigues. Yes. 

Mr. Shays. And they’re willing to commit on technology 8 at 
level 6. You’re not saying they’re willing to commit on technology 
4 at level 5, are you; or are you? 

Mr. Rodrigues. Right now our understanding is they plan to 
move ahead 

Mr. Shays. And every one of the levels I mentioned 

Mr. Rodrigues. Unless they trade these off. But right now 
they’re in the program — these are critical path technologies. These 
things are essential. Not technology 1 necessarily — technology 
areas 2 through 7 are critical to meeting the affordability goal. And 
if affordability is the primary factor driving this program, which is 
my understanding of what everybody is signed up to do, an afford- 
able next generation aircraft, then being able to launch or have a 
program without these becomes really problematic. 

The one example I used was if one of these areas was excluded 
or you had to go to its fallback, you would add almost 10 percent 
cost to the conventional variant which represents 1,700 plus of the 
aircraft they’re going to build. That is a significant cost difference. 
And that’s only one area. 

So these all deal with affordability. As long as affordability re- 
mains paramount, the question becomes if you don’t use these, 
what does it do to your cost projections? If you go do a block ap- 
proach which — talking about using an iterative process going with 
the block one that doesn’t have everything and moving on — I think 
we have to understand what is the implications of that. What 
you’re doing then is launching into the development of a product 
and the production of a product that I’m not sure it even comes 
close to meeting cost goals, because I don’t know what the effect 
of deleting these from the first blocks are. And once you have that 
production line started, if you’re betting on technology in some sub- 
sequent block in order to get the affordability into the program that 
you need in the long run, I don’t think that we solved anything by 
going to a block approach. 

Mr. Shays. I would agree with that. Let me just go back to your 
first chart again on the best practices model. There are three 
knowledge points. And I appreciate the indulgence of the commit- 
tee just to go a little further on this. Knowledge 1, this is basically 
the character of best practices act for any industry in this modern 
day and age. 

Mr. Rodrigues. That’s the best commercial practice. 

Mr. Shays. The concept is matches made between the customer’s 
requirements and the available technology. So whatever technology 
is available, how can we meet the customer’s requirements? We try 
to match those two. 

Mr. Rodrigues. Right. 

Mr. Shays. And then the second point is when the product’s de- 
sign is determined to be capable of meeting performance require- 
ments. 
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Mr. Rodrigues. Yes. And there’s a metric associated with that 
as well. The Joint Strike Fighter right now would be in the equiva- 
lent of what’s called the technology development phase. It is in risk 
reduction. There are concept demonstrations in the combined 
phase. So it’s really before that point of technology match. 

Mr. Shays. We’re not at point 2 yet. 

Mr. Rodrigues. No, the product development is the equivalent 
of the engineering and manufacturing development phase. Knowl- 
edge point 2 is a standard knowledge point that occurs both in in- 
dustry and in the Department. The Department just didn’t adhere 
to it. They actually have the standard and they have the metrics 
for measuring it. It’s something called CDR, critical design review, 
and the standard is that 90 percent of the drawings are released 
to manufacturing at that point. Now, unfortunately, the Depart- 
ment doesn’t adhere to that. The standard exists, they don’t apply 
it. In commercial industry, when we were doing the best commer- 
cial practice work, they exceed the 90 percent. 

Mr. Shays. I have two more basic questions. When we get to 
knowledge point 3, the product is determined to be producible with- 
in cost schedule and quality targets. Now, is the desire on the part 
of the military to move forward, in your judgment, with production 
before we have reached point 2, is that because they want the prod- 
uct to do more than right now the technology allows? 

Mr. Rodrigues. Let me try to clarify something. This isn’t about 
moving forward at this point with production. This is about entry 
into product development. 

Mr. Shays. Right. 

Mr. Rodrigues. And I don’t know 

Mr. Shays. But even the point of — the question still stands, and 
thank you for clarifying. The bottom line, though, is I’m just trying 
to understand the tension. In other words, they want this airplane 
to do more than the technology presently allows; correct? 

Mr. Rodrigues. Using technology readiness levels, that’s an ab- 
solutely true statement. 

Mr. Shays. So the issue is if they had to accept the technology 
that existed today, they wouldn’t be able to have this plane do 
what they want it to do. So that’s the tension. But they want the 
project to keep moving forward. The message that I’m hearing from 
GAO is saying, relax, we’re at the cutting edge, we’re not talking 
about the cold war where we have to rush this to the marketplace. 

Mr. Rodrigues. Absolutely. 

Mr. Shays. So we need to slow down, develop the technology, be- 
fore we start to do the development. 

Mr. Rodrigues. Yes. And the other thing they’re trying to get 
across in here is that when we make those decisions to accept these 
unproven technologies into a product development — and maybe I 
need to explain product development better. Product development 
is the engineering and manufacturing development phase in this 
context. In the engineering and manufacturing development phase, 
we should be focusing on developing and manufacturing the final 
product. In this case it’s the full-up plane with everything on it. 

What we do, or what we have done historically in the Depart- 
ment of Defense, is we go into that phase, which is difficult in and 
of itself if you’re using all proven technologies, and to try to inte- 
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grate a bunch of new technologies into a final product is still not 
an easy process. What we do is we allow immature technologies 
that are pacing items. These are defined, the ones we had up here, 
critical technology areas. Critical means they’re in the critical path 
to success. We allow that immature technology, unproven tech- 
nology, to enter into the product development phase. What you end 
up with then are long development cycles because now you have to 
bring those technologies along. You have people having to focus on 
technology development when we should be focusing on engineering 
and manufacturing development of a product, not the subtech- 
nologies that go into it. As I said, that’s a challenge by itself. 

And in industry they create a job that’s doable by a program 
manager. His job is to bring proven technologies together into a 
form that gives you the product that meets the customer’s require- 
ments. We expect our program managers to manage technology de- 
velopment and product development concurrently. And technology 
development is invention. Invention cannot be scheduled. And we 
set up tight schedules where these things have to fit in those 
schedules. The money is all lined up. Money becomes the driver, 
schedule becomes the driver. And what we have actually accom- 
plished tends to fall by the wayside. 

Mr. Shays. Let me recognize Mr. Blagojevich, the ranking mem- 
ber. I didn’t — if you have an opening statement you want to make 
I’m happy to have you do it, or we can get right into questions. But 
he’s your witness and you have him for as long as you need him. 

Mr. Blagojevich. Great. Thank you, Mr. Chairman I’ll just dis- 
pense with the opening statement and ask a couple of questions of 
Mr. Rodrigues. 

[The prepared statement of Hon. Rod R. Blagojevich follows:] 
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Opening Statement 

Representative Rod Blagojevich, Ranking Member 
Subcommittee on National Security, 
Veterans Affairs, and International Relations 

May 10, 2000 


GOOD MORNING. LET ME WELCOME OUR DISTINGUISHED 
WITNESSES FROM THE GENERAL ACCOUNTING OFFICE AND FROM 
THE DEPARTMENT OF DEFENSE, AS WELL AS THE WITNESSES ON OUR 
THIRD PANEL. I AM GLAD YOU ALL COULD BE WITH US TODAY. 

IT SEEMS THAT BOTH SIDES IN THIS DEBATE ARE CONCERNED 
ABOUT TIME AND MONEY, AS ARE WE. 

ON ONE HAND, THE DEPARTMENT OF DEFENSE WANTS TO 
CONSERVE BOTH BY UTILIZING INTEGRATED LAB AND GROUND 
TESTING RATHER THAN AN ADVANCED PROTOTYPE AIRCRAFT. THE 
DEPARTMENT CITES WITH PARTICULAR URGENCY THE AGE OF OUR 
CURRENT FLEET OF FIGHTER AIRCRAFT. 

G.A.O., HOWEVER, LOOKS AT HISTORY. BY MOVING TOO 
QUICKLY INTO ADVANCED PHASES OF DEVELOPMENT, D.O.D. HAS IN 
THE PAST BEEN FORCED TO HALT PRODUCTION TO ADDRESS 
UNFORESEEN PROBLEMS. HAD D.O.D. DEVELOPED ITS 
TECHNOLOGIES MORE THOROUGHLY TO BEGIN WITH, IT MIGHT 
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HAVE PRESERVED FUNDING AND ACHIEVED ITS ULTIMATE 
OBJECTIVES SOONER. 

GENERALLY SPEAKING, THERE MUST BE SOME POINT ALONG 
THE DEVELOPMENT SPECTRUM AT WHICH THE BENEFITS OF UP- 
FRONT PLANNING AND TESTING OUTWEIGH THE RISKS OF GOING 
FORWARD. 

THIS POINT - FOR THE JOINT STRIKE FIGHTER PROGRAM - IS 
REPRESENTED AS REQUIRING A "LOW LEVEL OF TECHNICAL RISK” IN 
CRITICAL TECHNOLOGIES BEFORE MOVING ON TO THE ENGINEERING 
AND MANUFACTURING DEVELOPMENT STAGE. 

ALTHOUGH MOST WOULD AGREE WITH THIS GENERAL 
STANDARD, THERE IS DISAGREEMENT ON ITS PARAMETERS AND 
REQUIREMENTS. THIS STANDARD COULD RANGE FROM TESTING 
TECHNOLOGIES INDEPENDENTLY, TO TESTING THEM TOGETHER IN 
CONTROLLED ENVIRONMENTS, TO INTEGRATING ALL 
TECHNOLOGIES IN A PROTOTYPE AIRCRAFT. 

THESE FACTORS ARE FURTHER COMPLICATED BY OUTSIDE 
INFLUENCES, AS WELL. POLITICAL REALITIES SUGGEST THAT 
DELAYS AND SHORT-TERM COST OVERRUNS COULD JEOPARDIZE 
SUPPORT FOR THE PROGRAM ITSELF. PROGRAM ADVOCATES WITHIN 
THE DEPARTMENT OF DEFENSE NATURALLY ARE HESITANT TO 
POSTPONE SCHEDULES, FEARING THAT SUCH A DELAY MAY BE 


2 



39 


MORE COSTLY TO THE PROGRAM AS A WHOLE THAN PROCEEDING 
WITH A SLIGHTLY HIGHER LEVEL OF TECHNOLOGICAL RISK. 

MR. CHAIRMAN. FOR THIS REASON, I THINK THIS HEARING IS 
VITALLY IMPORTANT. IT IS ABOUT MORE THAN JUST ACQUISITION 
REFORM IN THE J.S.F. PROGRAM. 

IT ALSO GIVES US AN OPPORTUNITY TO EVALUATE OURSELVES 
AND TO ANALYZE HOW CONGRESS VIEWS THE PROGRESS AND 
DEVELOPMENT OF ALL LARGE-SCALE D.O.D. PROGRAMS. 

I LOOK FORWARD TO THAT DISCUSSION. 

THANK YOU, MR. CHAIRMAN. 
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Mr. Blagojevich. Mr. Rodrigues, the chairman briefly men- 
tioned that if we slow down the effort to get into the engineering 
and manufacturing phase — that’s essentially the gist of what your 
argument is — that does not necessarily slow down the entire pro- 
gram. And it’s conceivable, is it not, that if we slow down in the 
technological development phase, that in the long run the time 
saved early on to get it right could actually shorten the process? 
Can you speak to that. 

Mr. Rodrigues. That’s exactly the fundamentals behind every- 
thing in the practices that we’re laying out from the work that we 
have done on best commercial practices: that putting the time in 
to get the technology match up front makes the product develop- 
ment cycle doable. Instead of having programs that extend for 10 
and 12 years, that are absolutely unmanageable — if I could share 
with you — would you put up the chart on the 10 years? This is 
what happens to you as programs stretch and grow — and this is 
DOD data. 

Now, I would contend that it’s very optimistic, but what their 
data show is that on average you’d have an 11-year program cycle. 
In 11 years you go through 4 program managers, 5 program execu- 
tive officers, 8 service acquisition executives, 8 defense acquisition 
executives, 5 chairmen of the Joint Chiefs of Staff, 7 Secretaries of 
Defense, 3 Presidents, and 11 cycles of coming up here to get 
money and going through the Department to get money. And we 
wonder why programs drag on and take forever? 

One of the problems we have is our cycle times becoming so long 
in the product development because we’re doing technology devel- 
opment and you should expect problems. We see them. I mean, this 
body approved entry into production on the F-22 with a single 
flight hour. And we expect that to stay on schedule? And we expect 
that to stay on cost? We knew virtually nothing about the capabil- 
ity of that plane in terms of demonstration. 

And what we have are a lot of hopes pinned on judgment about 
an ability to deliver those things. Judgment is not good enough in 
a best commercial practice. Demonstration is what counts. If you 
don’t demonstrate it, you end up with 12-year cycles and you end 
up with these problems and you end up with absolutely no account- 
ability. 

We have to match cycle time to program managers’ tenures to be 
able to hold people accountable. Give them a job that’s doable, 
which is product development when we’re in product development. 
And the only way you can do that is separating technology develop- 
ment from product development. And the only way you can legiti- 
mately do that is through a metric that allows you to absolutely 
measure demonstration. 

Mr. Blagojevich. I know that chart — three Presidents, seven 
Secretaries of Defense, four program managers, the only ones that 
last 11 years are generally the Members of Congress. 

Mr. Rodrigues. And, unfortunately, me. 

Mr. Blagojevich. And you. Can you talk about the importance 
of the continuity of keeping a program manager involved in a pro- 
gram like this, how it works in the commercial sector and how suc- 
cessful that is to have that continuity? 
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Mr. Rodrigues. Clearly, as I look at these and try to figure out 
what can you control here, the only one you could really control 
and would really have a direct effect is the program manager. Pro- 
gram executive officers have a whole portfolio; some of the others 
are political appointees, you can’t control that. But the program 
manager tenure can be controlled. And if you don’t have that, what 
you have is a situation we’re in now. The Joint Strike Fighter is 
on its third program manager. It’s only been around for 3 V 2 years. 
It’s on its third. The third one is there now. His job is to deliver 
this program into engineering and manufacturing development as 
scheduled. The money is there. They’re ready to go. Everything is 
lined up. It’s in all their outyear budgets. 

We talk about wonderful plans for modernization and the need 
to be able to get a new plane. And I agree with that. We need to 
get new planes out there. My problem with the way we’re going to 
do this one is it won’t get there in time. It won’t get there in quan- 
tity and we won’t reach modernization. This isn’t what’s going to 
get you there. 

In terms of matching that program manager’s tenure — and it is 
what gives you accountability, we have to give these people doable 
jobs. Put them in there for the tenure of the program to deliver the 
product so we can get focused on the product. Our focus now is on 
the next increment of funding, the next milestone. How can people 
stay focused on delivery of a product that’s 12 years away? I can’t 
think of anybody. To tell you the truth I have been doing this, lead- 
ing this work for over 12 years. I could not have focused 12 years 
ago on where I am today. There is no way. If you told me this is 
what I had to do, I would have told you you were crazy. It just 
doesn’t happen. You focus in shorter increments. People don’t think 
that way. 

Industry has gone from basically an 18-month to a 4 V 2 month 
cycle. The reason for that is to get that focus on product, give peo- 
ple doable jobs, get the stuff out there quickly because stuff turns 
over fast in terms of technologies. And rather than have these huge 
cycle times that lead to obsolescence — I mean, we’re spending hun- 
dreds of millions of dollars right now, on the F-22, to buy parts 
that are already obsolete. And we haven’t even built the plane. 
When you’re looking at 12- to 15-year cycle times, that’s what’s 
going to happen to you. We have to shorten that. 

I would submit as we did during a hearing before the Senate 
Arms Services Committee, Subcommittee on Readiness and Man- 
agement — that in developing a model that would deal with these 
issues. One of the things that you have to do is you have to put 
a strict limit on the engineering and manufacturing development or 
the product development phase. And 5 years, I think Dr. Gansler 
has said 67 months at this point, is their goal. Five years is a do- 
able thing. It allows you to match tenure. It brings accountability. 
It brings focus to the process. And it forces you to do trades on the 
front end. If you have a limit that you’re going to be held account- 
able to, then you will have to do the trades of the technology and 
costs on the front end that allows you to do a deliverable product 
in a 5-year cycle. 

Mr. Blagojevich. One final question, Mr. Rodrigues. We like the 
Joint Strike Fighter, you and I. You certainly — you have said you 
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do. I certainly think it’s a good idea. We’re for this. We want to see 
this get done sooner rather than later. It’s good for our military, 
good for our national defense. We agree to that. Right? 

Mr. Rodrigues. Right. 

Mr. Blagojevich. What do you say to the argument that if we 
adopt what you’re recommending as opposed to what the Depart- 
ment of Defense is recommending, this can hamper the ability to 
keep the support here in the Congress to be able to fund this pro- 
gram? 

Mr. Rodrigues. Unfortunately, I think that’s a real risk. I mean, 
it requires an understanding on the part of the Members. The fact 
of the matter is if we launch with immature technologies, you will 
not get this program on the schedule or on the cost. We will end 
up with the problems that we have had historically by going into 
product development with immature technology. You cannot sched- 
ule invention. It will create problems. And if they’re critical path 
items, they become the pacing item. And the cost of that is phe- 
nomenal. 

Let me give you some idea of this. Right now on the Joint Strike 
Fighter if you were to annualize numbers and take a look at it on 
a single contractor — because that’s where we’re going to go in 
EMD, down to one contractor — right now in the risk reduction 
phase, we’re spending the equivalent of about $265 million a year 
to do risk reduction. That includes the demo and all the other risk 
reduction activities on these critical technologies. To fail at $265 
million a year on something is kind of OK. When we go into EMD, 
our costs for a single contractor immediately jumps to an 
annualized rate of $1.2 billion and within 2 years of that we’ll be 
at $4 billion a year. 

Now, you have a problem with a critical path technology at that 
point you are carrying a $4 billion program on your back. Not $265 
million program, where you can tolerate some problems in inven- 
tion. And you know the basic cost tradeoffs are for every dollar 
mistake I make in the risk reduction or technology development 
phase, if I correct that mistake in the engineering and manufactur- 
ing phase it’s going to cost me $10, and if I wait until the produc- 
tion phase to correct that same problem — which the Department is 
doing on the F-22s without having not completed the development 
phase — here that $1 mistake now costs you $100 to correct. So this 
has to do with bringing affordability and discipline to the process 
to get what we need, which is a modernized fleet. And I just don’t 
see where following the practices of the past are going to get us to 
where we need to be. 

Mr. Blagojevich. Thank you, Mr. Rodrigues. 

Mr. Shays. Thank you. Mr. Terry. 

Mr. Terry. This is a very interesting discussion. Before I ask a 
couple of my questions I’ll say that I took a tour here a couple 
weeks ago of a company in my hometown that’s one of the leading 
high-tech support for our private sector industries across the Na- 
tion. I mean — and they made a comment during their presentation 
when I was being wowed by their technology. They said, “Our cli- 
ents expect us to be cutting edge but not bleeding edge.” 
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And as I’ve listened to your testimony and thoughts about imma- 
ture technology, that’s really what we’re talking about is getting 
ahead of ourselves in the technology and perhaps slowing down. 

I’ve got several questions but I’ll narrow it down to a couple here. 
And one is, just so I can put it in kind of a economical or simple 
terms, I see a catch-22 here where we in Congress expect out of 
our Department of Defense, and I think the people of this country 
expect if anyone is going to be on the bleeding edge or that’s ac- 
ceptable, it would be our Department of Defense. They want that 
type of technology in there. 

Now, assuming that they have those type of political pressures, 
how do they develop a more acceptable best commercial practice 
time line? It seems to me that they never can. How do we do that? 

Mr. Rodrigues. Right. 

Mr. Terry. Taking in the desire to assure the public that we’re 
using best commercial practices, but then also accepting the role 
that we want them to be on the far beyond cutting edge to maybe 
the bleeding edge of technology that’s always moving. 

Mr. Rodrigues. Right. I don’t see a disparity between these two 
things. What we’re talking about is how you go about getting a bet- 
ter product. It’s really about how do you get those technologies, 
those more advanced technologies in the field faster. Do you do it 
through a more iterative process or do you go for everything that’s 
out there. The latter takes us 20 years to get there and, by the 
time you get there, you can’t support it. It’s obsolete. You have got 
to buy out stuff because nobody builds this anymore. It really is 
about getting down to the basics of changing the culture and the 
incentives. 

But the other part of it is, you can be on the bleeding edge but 
be on it in technology development. Go chase those technologies. 
You need parameters around what you’re going to invest in, how 
you’re going to manage that technology base and how you’re going 
to graduate that technology to a product, and when is the right 
time to bring it into a product. I’m not saying don’t chase tech- 
nologies. I’m saying don’t chase it as part of a product development. 
That has cost way too much money. 

Mr. Terry. And hence, then, you’re doing your job in showing us 
or talking about the best commercial practices and how to institute 
that in the system. But then how do we justify, as Congressmen 
or the President or Department of Defense, you know, all right, we 
identify what we need in this plane, let’s manufacture it today in 
our Joint Strike Fighter force. Then the technology changes 2 or 4 
years after production. That makes it obsolete and we end up 
spending more money in a production of the next generation, soon- 
er than we probably should have. How do we justify that? I’m kind 
of asking you a political question. That’s not your realm here. 

Mr. Rodrigues. I think the Department is moving to — and prob- 
ably Mr. Soloway will talk about it — is the problem with the rapid 
changes in technology. It’s a reality. You can’t change that. People 
have to recognize that that’s the situation. And then what we real- 
ly need to do is go to a more iterative process in how we build 
things. In other words, we don’t decide today on a firm design that 
we’re going to build for 20 years. I mean, that isn’t how the world 
works. In fact, that isn’t how we really build planes. If you look at 
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the F-16, we built A’s and B’s and block 10 and block 20, block 30, 
block 40, block 50. Now we’re at block 60. Now we’re getting ready 
to build block 60-1 for somebody else. I mean, in reality that’s what 
you end up doing. Things change and you make those changes in 
the subsystems. But that basic design that you commit to has to 
last longer than just a few planes so you can get your money back 
out of it. 

So there are ways to do this. To me, it mostly rests with the De- 
partment. It’s what they come up here and sell. Unfortunately, we 
have a system now that incentivizes everybody to come up here 
and promise you the world and promise you all these wonderful 
things and lay out way early in a process an exact commitment to 
schedules and costs, build all that funding in. And you know when 
they’re doing that? They’re doing that before they even get into 
that technology development phase. They don’t have the slightest 
idea of whether those things are going to work or what they’re 
going to cost. But we lock into a commitment early on. 

And one of the things that needs to be done is to delay the com- 
mitment until you have the technology match. We shouldn’t allow 
people, to sell a product based on a technology that’s absolutely 
unproven. We’re raising everyone’s expectations to levels that are 
going to 

Mr. Terry. And during your presentation when you were talking 
about that and answering the chairman’s questions, I wrote a note 
down: So what responsibility does Congress have? Because I think 
we encourage that. So maybe we should ask you to do a GAO re- 
port on what type of congressional or political reforms we need in 
house so we don’t encourage that type. I agree that in trying to 
abide businesslike procedures, we don’t encourage them from our 
side to do it either. 

Mr. Rodrigues. We have done some pieces on this whole issue 
of what incentivizes the culture and the process at DOD, and Con- 
gress is a big player on that. 

Mr. Terry. Before we run and vote, let me ask this one simple 
question of you. In your testimony you had mentioned and use kind 
of colloquial language here, but they said — they’re talking about 
adopting the best commercial practices, they’re talking the talk but 
they aren’t walking it yet. How do we ensure that they are imple- 
menting these type of best commercial practices at the highest 
level instead of just talking about it? 

Mr. Rodrigues. I think you have to do it program by program. 
And the Joint Strike Fighter is the lead program and it is the one 
to do it on. And in the report I believe that you’re going to release 
here at the hearing, we have matters for congressional consider- 
ation. And those matters for consideration spell out what we think 
you should require the Secretary of Defense to lay out for you — and 
we need a metric that is understandable. 

Let me talk about technology readiness levels. They are actually 
very simple. I think that’s why people don’t like them because 
you’ll actually — you will understand them. And to simply say what 
they are, there are nine levels. What we’re talking about is pro- 
gressing to the level of 7 for entry into product development or en- 
gineering and manufacturing development. The first three levels 
are basically paper studies. 
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Now, I don’t know how many people want to launch a whole big 
aircraft program based on a paper study but I’m not too fond of it — 
paper study of a new technology. Not a good idea. The next two lev- 
els, 4 and 5 are basically laboratory hardware demonstrated in a 
laboratory environment. These are pretty easy things. I don’t know 
how many labs you’ve been out to, but I’ve been to laser labs and 
I was very disappointed. I thought I would see a laser. All I see 
is these parts spread all over tables. They’re hand wired. They can 
do the stuff you have to do in a lab. It’s only in that lab environ- 
ment. It’s not going to be flying on a plane like we want it to, or 
space aircraft. 

And then to get to level 6 and 7, you actually have to get that 
big thing that’s spread out in a lab down to the form and the fit 
that it will have to be in in the intended product. You have to have 
actually taken it out into the environment you’re going to use it in. 
That doesn’t mean you put it on the plane it’s going to be on, be- 
cause the plane doesn’t exist. That means you put it on a plane, 
if its something that has to fly, and you actually fly it. 

So now you have the form and fit that you’re going to need for 
the product, you put it into the environment — we’re not talking 
about operational testing. We’re saying understand the environ- 
ment. That’s level 6 and 7. These things are understandable. They 
are things that you can measure fairly readily. They are not sub- 
ject to engineering judgment or consensus, which is what we have 
tended to use or what we have used in the past that gets us into 
so much trouble. And they are things that you as a board of direc- 
tors need to begin to understand and to focus on and to hold the 
Joint Strike Fighter to. 

Mr. Terry. What is the timeframe for the product development 
timeframe? 

Mr. Rodrigues. Right now they’re planning on going into 

Mr. Terry. That you would recommend. 

Mr. Rodrigues. Oh, I would recommend that we limit it to 5 
years so we can bring accountability to the process. It would be 
hard with an 8 or 10-year cycle to say we’re going to put a program 
manager in there and we expect him to stay there for 10 years. I 
mean, life just doesn’t happen that way. 

Mr. Terry. Thank you. 

Mr. Shays. I’m sorry, we’re going to have to keep you here. We’ll 
have just a few more questions afterwards. We have to vote. We 
stand in recess. We’ll try and be back in 10 minutes. 

[Recess.] 

Mr. Shays. Mr. Rodrigues, the hearing is reopened. I’m curious, 
that took me 15 minutes? So next time I’ll say 15. I’m sorry. 

I want to just establish some basic points and I’m not looking for 
long answers. One of the points I want to establish is to under- 
stand what I think I understand but to just have you describe it. 
It’s not an uncommon practice for a building to be built before it’s 
been fully designed. And the argument is you can build it faster 
and you can save yourself from increased costs, inflation rate and 
so on. 

I make the assumption that when you’re building a building and 
you haven’t designed it, you still have all the technology there. So 
that kind of argument isn’t compatible where you kind of build be- 
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fore you’ve totally designed. Here we’re trying to make sure that 
we know that technology exists, obviously, before production, but 
even before development; is that correct? 

Mr. Rodrigues. Yes. Before product development, yes. 

Mr. Shays. And what I want to understand as well is that with 
the F-22, that basically is a new airplane, we’re not following this 
practice that we’re following with Joint Strike Fighter. The F— 18 
which we are — which the — E and F, and F is a modification, a larg- 
er F-18, but a modification of existing technology. It’s not a new 
plane, correct? 

Mr. Rodrigues. There’s very little commonality between the F— 
18C/D and the F-18E/F. The Department sold that as an ungraded 
F-18, but there’s nothing the same about it. 

Mr. Shays. So I didn’t get the answer I expected. But basically 
you consider it a pretty new plane. 

Mr. Rodrigues. Yeah. 

Mr. Shays. We’re building the B-22 but we made a decision that 
we would not build the AFX which was to replace the A-12 Navy 
attack plane. We dropped the multi-role fighter. So we are now 
going to be developing the JSF and it is going to have to — the JSF 
is going to have to fulfill a lot of very different roles. It’s going to 
be used by the Air Force in conventional takeoff and landing. It’s 
going to be used by the Navy for shorter takeoffs on carriers, and 
my understanding is it has to be a tougher plane and it’s going to 
weigh a little more. And then we are going to be using it for the 
Marine Corps and the U.K. Royal Navy as a vertical takeoff, like 
the Harrier jet. And yet it’s all coming from one program. So there 
are a lot of technologies that are in play here. 

Your bottom line point is that we should not develop and produce 
this plane until we follow the game plan, which is to make sure 
the technology exists and that we use best commercial practices. 
The first question that I want to ask in regards to best commercial 
practices, what best practices would not be appropriate in the DOD 
acquisition process? Can you think of any? 

Mr. Rodrigues. No I can’t think of any at all. Really what it 
then comes down to — and I think it was something that came out 
in an earlier question — it really comes down to how do you 
incentivize the process to put these things in place. What do we do 
to make it so that bringing knowledge to the table rather than 
judgment, to really be focused on setting a product development up 
for absolute success, how do you create the incentives to do that, 
to not oversell, to not overpromise, to not overcommit, but do what 
technology allows? 

Mr. Shays. How much cheaper is it to wait now to develop the 
technology than to begin to develop or produce this plane without 
the technology? 

Mr. Rodrigues. As I said, the rules of thumb are if you run into 
problems when you’re in technology development, $1 problem there 
becomes a $10 problem when you enter into product development. 
And if you actually get all the way to manufacturing or production 
and you’re still doing technology and you’re having problems, that 
same dollar problem now becomes a $100 problem. So 

Mr. Shays. So a 10-to-l ratio and then a 100-to-l ratio. 
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Mr. Rodrigues. Order of magnitude increases, yes. And some es- 
timates are much higher than that. Commercial industry, when 
you’re doing huge runs of production items where you have to do 
recalls, those numbers increase dramatically. 

Mr. Shays. Now, to the best of your knowledge — you’re speculat- 
ing — what would motivate, in your judgment, the military, the 
DOD to move forward? And if — I always believe there are logical 
reasons why they want to move forward. Do you think it’s the po- 
tential that they think Congress might withdraw if they’re not 
heavily committed, and therefore they’d rather get the plane this 
way, even if it costs more and they get less planes? Because if it 
costs more they’re going to get less planes. 

Mr. Rodrigues. Right. 

Mr. Shays. Or is there another reason that I’m not thinking? 

Mr. Rodrigues. No, I think that is it. It is the incentives. The 
money is in place. It’s very difficult to get money in wedges and 
approval and get something going. I mean, picture a program man- 
ager that comes in now. His job at this point in time on something 
like the Joint Strike Fighter, is to get the program into engineering 
and manufacturing development on schedule. The money is all 
lined up. In fact, the program manager leaves as the down select 
is made, and the process is handed over to somebody else who is 
going to have to worry about this. 

But let’s imagine that he takes a look at it and he says, well, 
wait a second, you know, some of these technologies we really don’t 
know all that much about it, so what I want to do is propose a 
delay. Well, now he has to go through the process of the Depart- 
ment of Defense where there are all kinds of people competing for 
these limited dollars waiting to find a place where they can take 
it from. And the Department does it itself all the time. Then it 
comes up here to the Congress and there are a lot of people looking 
to fund other things and there’s only so much money to go around. 
That risk is absolutely real. 

Mr. Shays. It’s a real risk but it’s even more of a risk if the F— 
22 and the F/A-18E and F are going to be much more costly than 
we anticipated because they’re not following these kind of practices. 
So in a sense their Joint Strike Fighter is going to be following a 
process that should save money in the long run or certainly not add 
to cost, competing with 2 other weapons systems that may gobble 
up costs. So I have sympathy for that if that’s it. 

Then I’ll just ask this last question. What is magical about a 
date, these time — I mean, is there — you haven’t developed any logic 
that says that a cold war enemy or a non-cold war enemy is going 
to be able to beat us and supercede us. So from that standpoint we 
don’t have a rush; correct? 

Mr. Rodrigues. Right. The objective was to get an affordable 
family of next generation aircraft. The key was affordable. That’s 
not basically threat driven at that point. 

Mr. Shays. I’m going to just say it can be refuted by DOD. But 
the bottom line is one of the potential luxuries of the cold war end- 
ing — I consider the world a more dangerous place but for other rea- 
sons — but the cold war has ended, we have some ability to develop 
the technology or a lot of ability to develop the technology before 
we go into development and production. 
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Mr. Rodrigues. Absolutely. But going back, if you don’t have the 
technology match, the fact that there is some threat out there you 
need to deal with doesn’t justify the mismatch of technology be- 
cause you’re going to run into problems when you try to build that 
thing. It is going to take you longer and longer and cost you more 
and more. 

Mr. Shays. So even if the cold war hadn’t ended, you would be 
arguing the same thing. 

Mr. Rodrigues. I would be arguing for some constraint in what 
you do as you move forward. There are ways to deal with that. 
There are some things where we don’t have capability — and I can’t 
talk about those where there are real threats out there that we 
can’t deal with, particularly in electronics — where we would go 
ahead with a very limited scale of something to get some kind of 
capability, even if we know it won’t meet the whole thing. It gives 
us something in the short run to be able to deal with part of the 
problem. There are cases where you can make a case for why you 
would want to take those risks. I don’t see those in the Joint Strike 
Fighter. 

Mr. Shays. Thank you. Mrs. Biggert has joined us. I yield to 
her — not yield to her but give her the floor. 

Mrs. Biggert. Thank you, Mr. Chairman. I just have one ques- 
tion. I’m sorry I missed the testimony. But yesterday the Senate 
Armed Forces Committee deleted the engineering and preliminary 
manufacturing money but said that the Pentagon could get the 
$424 million back if the plane proved it was ready to take the next 
step. Do you think that this will be a possibility? 

Mr. Rodrigues. That they’ll take the next step? 

Mrs. Biggert. Yes, that 

Mr. Rodrigues. That they’ll be able to accomplish that, take the 
next step within this year? Not really; no, I don’t. Not based on 
where they are in technologies, provided those technologies remain 
untradable from meeting the affordability goal. As long as those 
technologies are there — I don’t believe you could get it done in a 
year. Could they get lucky and do it? I suppose. I don’t think they 
have the demonstrations laid out at this point that would get them 
to the appropriate technology level to have that assurance. I just 
don’t see how that would happen. 

Mrs. Biggert. Thank you. Thank you, Mr. Chairman. 

Mr. Shays. Thank you. Mr. Terry. 

Mr. Terry. No. 

Mr. Shays. Is there any last — is there a question we should have 
asked that you would have liked us to ask or any point that you 
want to make? 

Mr. Rodrigues. There is one point that I skipped, actually two 
things. One is when we talked about the differences as far as we’re 
concerned in the scoring of the technologies and the Department’s 
position on that. I do want to make you aware that we had a closed 
hearing and we went through those technologies. 

And could you put that back up again? 

I want to point one thing out. One of the basic arguments was 
that there was a misunderstanding that when we scored the tech- 
nologies they thought we were saying in order for it to get to a 
level 7, the level needed to enter EMD, that it had to be on the 
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Joint Strike Fighter aircraft. As I said, that would be an impos- 
sibility, obviously not the standard. We spent a great deal of time 
working with the people who apply this at the contractor’s plants 
and the program office to get this clear understanding that we 
were talking about demonstrating something in close form and fit 
in the environment; not on the Joint Strike Fighter, on some kind 
of surrogate platform. 

But if you look at this, you can see a number of those tech- 
nologies are projected only to be at readiness level 4; technology 
area 7, technology area 3, and technology area 2. As I said earlier, 
level 4 is not form and fit, it is laboratory hardware in a laboratory 
environment. So this argument about, oh, there was confusion 
about scoring this because we thought you meant you had to have 
it on a Joint Strike Fighter doesn’t even come into play in the scor- 
ing of those technologies. 

And I can tell you we made that absolutely clear. So there is no 
misunderstanding. 

The other thing is what can Congress do or what should you do? 
I think we should require them to be held to a level 7 of dem- 
onstration. And in those cases where they can or believe that they 
should move without that, they should have to provide a very dis- 
creet — first of all, that they score it properly and own up to the 
scoring. Let’s not play games. These are pretty clear things to 
score. There is no confusion. 

And once we score it, if the scoring comes out less than a level 
7, the only way they should be able to move forward is make it 
clear to you where they are on those critical technologies and then 
explain to you why it is that we need to take on that additional 
risk of moving forward without having demonstrated the tech- 
nologies that are going to be critical to the building of that final 
product, taking those technologies and having the very difficult 
task of trying to integrate a whole bunch of technologies into a 
final product. 

Mr. Shays. You wanted to make that point, but it does raise a 
question now. And the question it raises for me is, is this a package 
deal? Do all have to be there in order to go to that next step of de- 
velopment or do we isolate each one of these technologies as a sep- 
arate issue before we move forward? 

Mr. Rodrigues. As long as they’re on the critical path, the pac- 
ing item in any area, whatever it is — the long pole in the tent is 
what you have to worry about. 

Mr. Shays. Listen to the question I’m asking. Are they independ- 
ent or do I have to take them all as a package? 

Mr. Rodrigues. No, they’re independent. You do each one. 

Mr. Shays. Right. I anticipated that would be the answer but I 
just wanted to make sure. 

Are you at liberty to tell us which areas have the greatest chal- 
lenge right now? Is this a proprietary issue or not? I can’t imagine 
it would be. Without identifying which is one and which is two and 
so on, can you tell us the area where we’re doing the best and 
where we’re potentially doing the worst? 

Mr. Rodrigues. Clearly, the best 

Mr. Shays. Let me ask a question. Is that an uncomfortable 
question? 
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Mr. Rodrigues. I can say clearly from this, the best are the ones 
that would be technology area 1 which I think you have a thing 
that tells you what that is. I can’t — I don’t want to say. 

Mr. Shays. Fair enough. 

Mr. Rodrigues. But clearly one and eight. 

Mr. Shays. Someone will have to explain to me, and I should 
have clarified before but I’m not going to push the point, why it 
even matters, why we can’t have that dialog. But, fair enough. I 
guess because we haven’t made a choice on who gets what. 

Mr. Rodrigues. Both contractors don’t work the same tech- 
nologies. 

Mr. Shays. That’s fair. You’ve been wonderful. You’ve put the 
ball in play and you’ve given an opportunity for those that follow 
to answer your points. I don’t know if you or someone else can stay 
to hear the other presentations because we might seek to have your 
office respond. You preferably, but someone else. 

Mr. Rodrigues. I’ll stay. 

Mr. Shays. We’ll go to our second panel. 

Is that OK? 

We’ll go to panel two. And I thank you, Mr. Rodrigues. 

We have Mr. Stan Soloway who is the Deputy Secretary of De- 
fense Acquisition Reform, Department of Defense, and Major Gen- 
eral Raymond Huot, U.S. Air Force Acquisition Program, Depart- 
ment of Defense. If you both would stand, we’ll swear you in. What 
I think I will do is just slide you over a little bit. Mr. Soloway, if 
you could just move your chair a little bit so we give Mr. Huot — 
General Huot, I’m sorry General. If you raise your right hands, 
please. 

[Witnesses sworn.] 

Mr. Shays. Thank you. Gentlemen, it’s great to have you here. 
I appreciate your spending the time with us. And I want to make 
sure that, Mr. Soloway, we have you first. Am I breaking protocol 
or am I keeping protocol? 

Mr. Soloway. No, that’s correct. 

Mr. Shays. I’m in charge. Well, Mr. Soloway, we’ll start with 
you. Thank you for being here. 

STATEMENTS OF STAN Z. SOLOWAY, DEPUTY SECRETARY OF 

DEFENSE, ACQUISITION REFORM, DEPARTMENT OF DE- 
FENSE; AND MAJOR GENERAL RAYMOND HUOT, U.S. AIR 

FORCE, ACQUISITION PROGRAMS, DEPARTMENT OF DE- 
FENSE 

Mr. Soloway. Thank you, Mr. Chairman and members of the 
committee. It’s a pleasure for me to be here today. If I could just 
divert for a moment from my prepared text and say that I do be- 
lieve that there is extraordinary commonality between what the 
GAO is recommending and what we are looking at in terms of a 
strategy of how to deal with the issues before us, and hopefully 
we’ll have a robust discussion of that as we move forward. 

But I am pleased to be here to have this opportunity to discuss 
with you our continued progress with acquisition reform and par- 
ticularly how it relates and applies to the Department’s Joint 
Strike Fighter program. 
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As you know, acquisition reform has been a top priority for the 
Department for the last several years and encompasses a wide 
range of initiatives and has had many real successes. Let me list 
just a few. One excellent example is the Joint Direct Attack Muni- 
tion [JDAM], which performed so flawlessly in Kosovo. Designated 
as an acquisition pilot program, JDAM was originally expected to 
cost in excess of $40,000 per unit, but through a combination of ac- 
quisition reforms and focused innovative program management we 
can now purchase JDAM for less than half that amount. 

Mr. Shays. That’s the unmanned plane. 

Mr. Soloway. No, JDAM is essentially a guidance, is the best 
way to describe it. 

General Huot. Probably the best way to say it, it’s a strap-on kit 
for a general purpose bomb. It gives it all weather capability. INS, 
Inertial Navigational System, with global positioning system guid- 
ance. 

Mr. Shays. Gotcha. I’ve seen it and I appreciate having it clari- 
fied. 

Mr. Soloway. Then there is the precision location GPS receiver 
or PLGR. This receiver, purchased largely through the new com- 
mercial buying authorities contained in the Federal Acquisition 
Streamlining Act, replaces a previous field version built to exten- 
sive military specifications that weighed over 30 pounds, required 
two operators, had only one channel and cost us thousands of dol- 
lars per unit. PLGR, on the other hand, requires only one operator, 
has five channels, weighs just over 2 pounds and costs less than 
$1,000 per unit. 

The All Ordnance Destruct System [AODS], is a flight termi- 
nation system used on the rocket system launch program vehicles 
at the Space and Missile System Center. The majority of these ve- 
hicles are used as targets in support of the ballistic missile defense 
programs. By utilizing the commercial buying authority known as 
FAR, Part 12, unit costs were reduced from the previous purchase 
price of $900,000 a kit down to $55,000 a kit, but more impor- 
tantly, the new kits are also technically superior. 

Today, circuit cards for the avionics in the F-22 are being pro- 
duced largely on a commercial line at TRW, thus saving the De- 
partment significant resources that would have had to be devoted 
to unique development and production facilities. Moreover, the reli- 
ability testing on those circuit cards have demonstrated excellent 
results and costs appear to be significantly lower than expected, 55 
percent to 70 percent less than their military standardized counter- 
part. 

Acquisition reform is also central to the development of the 
Navy’s new Virginia Class Attack Submarine. Key to the success 
of the program has been the use of integrated product and process 
development teams, the use of open systems architecture, and the 
insertion of commercial off-the-shelf technologies. The Navy will 
benefit there with a cost avoidance of $30 million per ship but, 
more importantly, the Virginia Class will operate at a 32 percent 
lower total ownership cost than the comparable Seawolf. 

Indeed, much has changed and for the better. Given the complex- 
ity of our business practices and the entrenched cultures we have 
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inside and outside of the government, I believe we have made sub- 
stantial progress. 

But clearly we must do more. The security environment we face 
is unpredictable and unstable. And our success in meeting the chal- 
lenges of the battlefield of the near and more distant future will 
hinge in large part on our ability to access and integrate true cut- 
ting-edge technologies that provide us the dominance, speed, and 
scope of information that we need. 

One of the means for accomplishing these goals is through a re- 
structuring of how we develop, manufacture, and maintain our 
weapon systems. It is no secret that cycle times for new weapon 
systems from concept to fielding remain unacceptably high and 
that such long cycle times too often result in the fielding of already 
obsolete technologies. Since some technology decisions must be 
made early in a program, it is clear that our history of taking 15 
or more years to field new systems is not at all consonant with the 
torrid pace of technology change we see today. 

There are, of course, many reasons for these long cycle times. 
Key among them is often the very nature of the requirements set 
forth for any individual program. Traditionally, our requirements 
have been both inflexible and involved extraordinary technology 
challenges that can take many years to meet. This is beginning to 
change. Today both our operational and acquisition and technology 
communities recognize that to optimize support for our men and 
women in the field and to most responsibly steward the public’s tax 
dollars, we need to institute new requirements and acquisition 
strategies. Indeed, we are confident we can significantly reduce 
cycle times and costs and provide items for the warfighter faster, 
for a more flexible evolutionary approach. 

How we do this, how we develop, manufacture, and maintain sys- 
tems is based on the Department’s 5000 Series documents, our 
bible for systems acquisition, which we are currently rewriting. The 
DOD 5000 rewrite will drive the Department further toward evolu- 
tionary acquisition and increase our focus on flexibility and re- 
quirements. Additionally, the new 5000 will require greater tech- 
nology maturity prior to entering the manufacturing phase of a 
program. 

In the new systems acquisition environment, key acquisition de- 
cisions and long-term funding commitments may not be made until 
technology shows the required maturity and risks are better under- 
stood and mitigated than has traditionally been the case. And the 
JSF program is a forerunner of this new approach. Indeed, since 
its inception, the JSF program has been recognized for utilizing 
and actually pioneering many acquisition reform concepts and ap- 
plying them to the actual business processes and contract vehicles 
being utilized, including but not limited to the critical precept of 
the new 5000 series. 

For instance, modeling and simulation has been proven in both 
industry and government to help reduce the time resources, and ul- 
timately risk, associated with systems development. Representa- 
tions of proposed systems, basically virtual prototypes, are embed- 
ded in realistic, synthetic environments to support the various 
phases of the acquisition process. The JSF program has made ex- 
tensive use of M&S in the requirements development process and 
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is continuing the use of M&S with mission level virtual pilot-in- 
the-loop simulation to support a more thorough evaluation of re- 
quired avionics capability. 

In keeping with the best proven practice in the commercial tech- 
nology world, virtual manufacturing and virtual maintenance are 
also being pursued to facilitate planning and drive down the associ- 
ated cost. 

The JSF is also incorporating an evolutionary acquisition strat- 
egy. In this process, the warfighter and the buyer work side by side 
to facilitate a better understanding of the requirements and deci- 
sions on tradeoffs between performance and cost. Specifying oper- 
ational requirements in an incremental manner phased over time 
and matching them against the projected threat and available tech- 
nologies have allowed the JSF program to exercise thoughtful judg- 
ment in balancing cost, schedule and performance. 

The use of evolutionary acquisition further mitigates risks by al- 
lowing technologies to be inserted as they mature. 

As I mentioned before, probably the most important change in 
the new 5000 rewrite is the emphasis on technology maturity be- 
fore entering into system integration, or what is today known as 
engineering and manufacturing development. Of course, transition 
into EMD is a challenge in every program. And in the JSF pro- 
gram, as in others, it will be up to the design teams and the pro- 
gram office along with the Service Acquisition Executives and the 
Defense Acquisition Executives to determine the acceptable level of 
technology readiness prior to EMD decision. 

Among the many factors that can help us make overall tech- 
nology readiness assessment are technology readiness levels 
[TRLs], which are used sometimes in the DOD. The minimum TRL 
rating of 1 begins at paper studies, as Mr. Rodrigues said, of a 
technology’s basic properties and rises to the maximum rating of 9 
for a system in its final form, operating under mission conditions. 

However, there is no hard and fast rule in DOD and NASA or 
elsewhere in government as to specific threshold TRLs for any 
given decision. Additionally, it should be noted that there are more 
comprehensive methodologies that we do use which can and do pro- 
vide more value as risk management as opposed to risk measure- 
ment tools. Indeed, the DOD 5000 rewrite does not prescribe a re- 
quired technology readiness level but does recommend using TRLs 
as a tool to help measure the maturity level of the technology. 

What the 5000 does prescribe is that technologies be dem- 
onstrated in a relevant environment with a fallback plan at a high- 
er maturity level. In other words, if a far-reaching, newer tech- 
nology does not pass relevant testing, a lesser proven technology 
could be utilized as long as it enables the system to still achieve 
its critical performance requirements. 

Recently the GAO provided a draft report to the Department on 
the JSF that recommends extending the JSF development schedule 
to allow for further maturation of technologies. Their recommenda- 
tion is based on their understanding that critical technologies will 
have inadequate levels of technology maturity based on TRLs at 
the time the EMD contract is to be awarded and the decision made 
in the spring of 2001. 
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The GAO report clearly articulates a strategy for systems devel- 
opment that we embrace, as evidenced by the revisions to the DOD 
5000 I mentioned earlier. Indeed, for the most part, we are in vio- 
lent agreement. Where we differ is on the definition and applicabil- 
ity of TRLs in the decisionmaking process. GAO’s position does ap- 
pear to be that achieving a TRL level of 7 should be required prior 
to entering EMD, again as Mr. Rodrigues pointed out. Our view, 
however, is that a level 7 requires the very kind of systems integra- 
tion that takes place during EMD and that it is infeasible to 
produce full-scale testing of this type prior to that phase. 

Moreover, as noted earlier, we, like NASA and others, see TRLs 
as but one input to the decision process. In fact, as the NASA Di- 
rector of Programs told us, “NASA does not formally use nor rigor- 
ously apply the definition of technology readiness level, TRL, to its 
systems development. We generally commit to development of oper- 
ational systems when all critical technologies have achieved TRLs 
of 5 or 6.” 

In assessing a technology prior to EMD, we do seek to assess 
whether the individual technology has been proven or is close to 
being proven in appropriate developmental environments. We do 
not agree that it is necessary or even feasible to demonstrate the 
full integration of the technology prior to EMD. Indeed, the GAO 
report does seem to require that all technology must be flown on 
an actual or prototype JSF platform in order to demonstrate ade- 
quate maturity for EMD. However, much of the JSF avionics and 
software, for example, can and should be demonstrated on the 
ground at a lower cost. Technologies that must be flown for ade- 
quate demonstration will in fact be or have been flying on the con- 
cept demonstrator aircraft, commercial aircraft, the F— 16, the F— 
22, the F/A-18E/F and the Eurofighter. 

So, in short, although the technology will not be demonstrated on 
the actual JSF or a JSF prototype per se, it will be tested in a rel- 
evant environment. And the success of those demonstrations is crit- 
ical to our confidence in our ability to successfully integrate the 
technologies during EMD. And we are confident we will have all 
critical technologies at an adequate and appropriate technology 
readiness level by the time an EMD decision must be made next 
year. I emphasize the decision is not to be made for another year. 

GAO has also expressed concern that if we make a premature 
production decision we could be locked into manufacturing proc- 
esses based on an expected technology capability, thus creating the 
risk that if the integration fails we will face exorbitant cost and 
time in redevelopment. 

We agree there is risk. There is always risk in the integration 
of complex systems. And how the JSF program mitigates those 
risks is the key. As I think you will hear from Major General Huot, 
the risk mitigation initiatives associated with JSF have probably 
been the most comprehensive and aggressive of any DOD program 
ever. 

Further, as I noted before, the rapid maturation of modeling and 
simulation capabilities has enabled the development and testing of 
a wide range of critical technologies that might not otherwise have 
been possible and has played a key role in our ability to effectively 
assess the risks on the JSF program. Thus, any decision to proceed 
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into EMD and production of the JSF will carefully assess all risks 
against the fallback alternatives. 

Finally, let me be very clear. The strategy of revised acquisition 
process that will be prescribed in the DOD 5000 rewrite and which 
is largely being implemented on the JSF does represent a very real 
departure from our traditional approach to systems development. 
In the past, we would indeed consider technology development as 
part of the engineering and manufacturing phase. Now we have 
technology development before we make a commitment to — proceed 
into system development and demonstration, and eventually pro- 
duction. 

Mr. Chairman, acquisition reform has been made possible by a 
strong partnership between the Congress and the executive branch, 
and reflects our mutual commitment to ensuring that the govern- 
ment operates far more efficiently and effectively than has histori- 
cally been the case. I appreciate having had this opportunity to be 
here today, because as we continue on the successful path we have 
forged, we need the continued support and commitment of the Con- 
gress. 

We also appreciate the continued support and encouragement the 
GAO has provided as we continue to change the worlds’s largest 
buying institution. Our disagreement on the specific role of TRLs 
notwithstanding, we are in close agreement on where the DOD’s 
acquisition process must go. We also believe it is important to let 
programs like JSF that are demonstrating the acquisition and 
management reforms of recent years to have the flexibility to man- 
age their programs and make decisions based on weighing the risks 
against costs, schedule, and technology maturity. 

That concludes my statement. I would be happy to answer any 
of your questions now and certainly stand ready to provide any ad- 
ditional information the committee believes would be helpful in fos- 
tering a clearer understanding of this or other DOD programs. 

Mr. Shays. Thank you Mr. Soloway. That was a very helpful 
statement. We did allow you to go over the 10 minutes because I 
think it’s important for you and DOD to really state your case on 
the record. Then we can examine your statement. 

[The prepared statement of Mr. Soloway follows:] 
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Mr. Chairman, members of the committee, it is a great pleasure for me to be 
here today, and I thank you for this opportunity to discuss with you our continued 
progress with acquisition reform, particularly how it applies to the Department’s 
Joint Strike Fighter (JSF) Program. Before I get into the specifics on the JSF, I 
would first like to begin our discussion with an understanding of the changes we 
have made and continue to make throughout the Department. 

As you know, Acquisition Reform has been a top priority for the department 
for the last several years, and encompasses a wide range of initiatives. From 
eliminating unnecessary government unique process requirements in favor of 
commercial performance standards.. .to using innovative contracting approaches 
that allow us to access commercial technology providers that would otherwise be 
unable, and unwilling to do business with us. Key to the progress we have made has 
been our strong partnership with Congress. 

Many of our initiatives are specifically designed to allow more flexibility in 
managing programs and increasing business opportunities with the continuing 
overarching goal of providing systems, equipment, services and other support 
faster, BETTER and less expensively, to our men and women in uniform. We have 
had many, real successes in implementing Acquisition Reforms. Let me mention 
just a few programs that illustrate the progress made. 

One excellent example is the Joint Direct Attack Munition (JDAM), which 
performed so flawlessly in Kosovo.. ..designated as an acquisition pilot program, 
JDAM was originally expected to cost in excess of 840,000 per unit; but through a 
combination of acquisition reforms and focused, innovative program management, 
we can now purchase JDAM for less than half that. 

Then there is the Precision Location GPS Receiver, or PLGR...lhis receiver, 
purchased largely through the commercial buying authorities contained in the 
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built to extensive military specifications, that weighed over 30 pounds, required two 
operators, had only one channel and cost us thousands of dollars per unit...PLGR 
on the other hand, requires only one operator, has five channels, weighs just over 
two pounds. ..and costs less than a thousand per unit. 

The All Ordnance Destruct System (AODS) is a flight termination system 
used on the Rocket System haunch Program (RSIJP) vehicles at the Space and 
Missile Systems Center Test and Evaluation Directorate. The majority of these 
RSLP vehicles are used as targets in support of Ballistic Missile Defense Programs. 
By utilizing FAR Part 12 commercial contracting methods, unit costs were reduced 
from the previous purchase price of $900K per kit, down to $55K per kit. Not only 
were they cheaper, but the new kits are also technically superior - the new' modular 
design permits multiple configurations, and eliminates unnecessary components 
(such as batteries and the aft launch umbilical cable). 

Today, circuit cards for the avionics in the F-22 are being produced largely 
on a commercial line at TRW... thus saving the Department significant resources 
that would have had to be devoted to unique development and production facilities. 
Moreover, the reliability testing on those circuit cards have demonstrated excellent 
results. ..and costs appear to he significantly lower than expected, 55%-70% less 
than their military standardized counterpart. 

Getting away from military unique requirements, where possible, is our goal. 
For instance, aircrews must have boots that will protect them in the often 
hazardous, military unique environment. In preparing to buy a new boot for our 
aircrews, the Defense Logistics Agency (DLA) and the Air Force first did extensive 
market research to locate commercial sources. Candidate boots were then subjected 
to fire resistance and thermal protection testing to simulate the stressful 
environment the boots would have to endure. Historically, the procurement process 
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for new, military unique equipment takes 7 to 10 years. In less than a year, DLA 
and the Air Force were able to purchase this new, commercially manufactured boot. 

Acquisition Reform is also central to the development of the Navy’s New 
Virginia Class Attack Submarine. Key to the success of the program has been the 
use of Integrated Product and Process Development Teams that ensure the most 
efficient design early in the development process. The use of open systems 
architecture and insertion of COTS technologies has resulted thus far in savings of 
$24 million. By simplifying specifications, using appropriate commercial standards 
and reducing the number of drawings required by the government, the Navy will 
benefit from a cost avoidance of $30 million per ship. More importantly, the 
Virginia Class will operate at a 32% lower total ownership cost then the comparable 
SEAWOLF. 

Indeed, much has changed. ..and for the better. Given the complexity of our 
business practices, and the entrenched cultures we have inside and outside of 
government, I believe we have made substantial progress. But clearly, we must do 
more. 


The security environment we face is unpredictable and unstable, and 
includes unprecedented threats of chemical and biological warfare, and the 
continued threat of nuclear confrontation. Our nation’s military leaders have set 
forth, in Joint Vision 2010, their blueprint for meeting the principal challenges of 
the next decade and beyond. Joint Vision 2010 makes clear that our success in 
meeting those challenges will hinge, in large part, on speed and information. ..and 
our ability to access and integrate true cutting edge technologies that prov ide us the 
dominance, speed and scope of information we need. 

To achieve these goals, the Department is focused on a number of high 
priority initiatives: fielding high quality products and systems quickly and 
supporting them responsively; reducing the total ownership costs of new and legacy 
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systems; and reducing the overhead burdens imposed by our current acquisition 
and logistics infrastructure. One of the means for accomplishing these goals is 
through restructuring how we develop, manufacture and maintain our weapon 
systems. 

It is no secret that cycle times for new weapons systems — from concept to 
fielding— remain unacceptably high and that such long cycle times too often result 
in the fielding of already obsolete technologies. Since some technology decisions 
must be made early in a program, it is dear that our history of taking 15 or more 
years to field new systems is not at all consonant with the torrid pace of technology 
change that we see today. There are, of course, many reasons for these long cycle 
times. Key among them is often the very nature of the requirements set forth for 
any individual program. Traditionally, our requirements have been both inflexible 
and involved extraordinary technology challenges that can take many years to meet. 
Indeed, it has been suggested that we might spend as much as 30% of a weapon 
system’s cost in our efforts to meet just the last 10% or 15% of performance 
requirements. And even when a system is finally fielded, as noted above, some of its 
components are sometimes already obsolete. 

This is beginning to change. Today, both our operational and acquisition & 
technology communities recognize that to optimize support for our warfighters in 
the field, we need to institute new requirements and acquisition strategies that 
include more flexibility. This added flexibility will allow the kind of logical 
tradeoffs, based on demonstrated technologies and cost, that will enable us to field 
new capabilities more quickly. By using what we call an evolutionary approach, we 
are confident we can indeed significantly reduce cycle times — and costs — and 
provide systems to the warfighter faster. 

Through evolutionary acquisition strategics we can define, develop, produce, 
and deploy an initial militarily useful capability based on the best proven technology 
available, time-phased requirements, projected threat assessments, and 
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demonstrated manufacturing capabilities. The scope, performance capabilities, 
and timing of future “Blocks” would thus be based on continuous communications 
between the requirements, acquisition, intelligence, science and budget 
communities, and the enhanced maturity of the technology involved. 

To facilitate an acquisition strategy that is evolutionary, the program 
manager must use the appropriate enabling tools - the incorporation of open 
systems and suitable risk assessment - to match the time-phased requirements to 
critical needs and available technologies. Open systems is a design strategy of 
incorporating existing, widely accepted standards for system interfaces that allows 
technology insertion without requiring a major redesign - a “plug and play” 
concept. And part of determining when a new technology will be incorporated into 
the design, is, of course, considering the maturity of the technology. 

How we do this.. .how we develop, manufacture and maintain systems, is 
based in the Department’s 5000 Series documents. Currently, the OSD and Service 
staffs are engaged in rewriting this principal policy guidance — our “bible” for 
systems acquisition - the DoD 5000.1, The Defense Acquisition System, and the DoD 
5000.2, Operation of the Defense Acquisition System. 

The Joint Chiefs of Staff have already completed a rewrite of the Chairman’s 
“3170 Series Requirements Generation System” to reflect more flexible and time- 
phased requirements, interoperability as a key performance parameter, and 
affordability in requirements documents. The DoD 5000 rewrite will, like the 
changes to the Joint Chiefs of Staff instruction, address evolutionary acquisition and 
focus on flexibility in requirements. Additionally, the new DoD 5000 will require 
greater technical maturity prior to entering the manufacturing phase of a program, 
integrate acquisition and logistics early in the process, and more. In the new 
systems acquisition environment, key acquisition decisions and long term funding 
commitments may not be made until technology shows the required maturity and 
risks are better understood and mitigated, than has traditionally been the case. 
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One of our programs embracing Acquisition Reform and this new approach 
to the front end of the acquisition process, is the JSF. Since its inception, the JSF, 
known as the Joint Advanced Strike-Fighter Technology (JAST) in it’s pre- 
acquisition days, has been recognized for utilizing and actually pioneering many 
Acquisition Reform concepts, and applying them to the actual business processes 
and contract vehicles being utilized. 

For instance, prior to the JSF Program, there was little experience and 
capability within DoD to perform electronic source selection for major, high 
technology research and development activities. The old way of doing business 
required the contractor to deliver his proposal literally by the truckload. In the first 
source selection, two contractors were chosen to participate in the 4-year concept 
demonstration phase, which is currently ongoing. In this initial competition, the 
JSF Program replaced the traditional paper process with an electronic process, thus 
reducing the total time from initial planning through award from the normal 12 
months to 4 months. This early success in electronic commerce gave the JSF 
Program the confidence to implement further acquisition reform initiatives. 

Consistent with Acquisition Reform and the Department’s policy, 
performance based specifications are the JSF Program’s goal. True to this ideal, 
the philosophy of the JSF program is to tell industry what is needed, not how to meet 
the need. The Request for Proposal (RFP) for the Concept Demonstration Program 
contained no minimum requirements, and instead gave the contractors the freedom 
to trade off cost and performance to achieve the most cost-effective solution. In 
addition, the contractors were encouraged to utilize commercial standards and best 
practices and to make recommendations regarding the use of specifications. While 
military specifications do have value, they are often an unnecessary burden to the 
contractor, frequently have an even higher performance commitment than 
required, and impart other non-value-added requirements and tremendous 
documentation. 
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Simulation Based Acquisition is also a major thrust of the Department and is 
one of the key tenets of the DoD 5000 re-write. Modeling and Simulation (M&S) has 
been proven in both industry and government to help reduce the time, resources 
and ultimately, risk associated with systems development. Representations of 
proposed systems, virtual prototypes, are embedded in realistic, synthetic 
environments to support the various phases of the acquisition process. The JSF 
Program has made extensive use of M&S and Cost as an Independent Variable 
(CAIV) trade studies in the requirements development process. JSF is now 
continuing the use of M&S with mission level virtual pilot-in-the-loop simulation to 
support a more thorough evaluation of required avionics capabilities. Virtual 
Manufacturing and Virtual Maintenance are also being pursued to facilitate 
planning and thereby drive down the associated costs. 

The JSF is also incorporating an evolutionary acquisition strategy, as defined 
earlier in my statement. The program is truly joint and has achieved an 
unprecedented level of cooperation between industry and the participating U.S. and 
international services, and between the requirements and development 
communities. This cooperation enables the communication flow* that is the key to an 
evolutionary acquisition strategy. In other words, the warfighter and the buyer 
work side by side to allow a better understanding of the requirements, facilitating 
decisions on trade-off between performance and cost. Specifying operational 
requirements in an incremental manner phased over time, and matching them 
against the projected threat and available technologies, have allowed the JSF 
Program to exercise thoughtful judgement in balancing cost, schedule and 
performance. And JSF’s use of evolutionary acquisition further mitigates risks by 
allowing technologies to be inserted as they mature. 

As I mentioned before, probably the most important change in the new DoD 
5000 re-write is the emphasis on technology maturity before entering into system 
integration, or what is known as Engineering and Manufacturing (EMD) in today’s 
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DoD 5000, Approximately one quarter of JSF pre-EMD resources are allocated to 
technology maturity efforts in the crucial technology areas of flight systems, 
propulsion, supportability and training, structures and materials, manufacturing 
and produce-ability, and mission systems. Technology investments have been 
selected based on the potential to improve life cycle affordability and enhanced 
capability in the areas of survivability, supportability and lethality. 

Transitioning advanced technologies from the laboratory into operational 
systems is a challenge in every program. In the JSF Program, the design teams and 
the program office will determine the acceptable level of technological readiness. 
Operators, maintainers and the design teams have been involved in the selection and 
structuring of technology maturity projects from the beginning. This has resulted 
in an improved coordination between historically distinct laboratories, system 
design/development, and user communities. 

Among the many factors that help us make overall technology readiness 
assessments are Technology Readiness Levels, or TRLs. The minimum rating of 
“1” begins at paper studies of a technology’s basic properties and rises to the 
maximum rating of '‘9’’ for a system in its final form operating under mission 
conditions. However, there is no hard and fast rule, in DoD, NASA, or elsewhere, as 
to specific threshold TRLs for any given decision. Indeed, the DoD 5001) rewrite 
does not prescribe a required level of technology readiness, but instead recommends 
using TRLs as a tool to help assess the maturity level of the technology. What the 
DoD 5000 does prescribe, is using technologies demonstrated in a relevant 
environment, and having a fallback plan at a higher maturity level. In other words, 
if a far-reaching newer technology did not pass relevant testing, a lesser, proven 
technology could be utilized, as long as it enables the system to still achieve its 
critical performance requirements. 

Recently, the GAO provided a draft report to the Department on the JSF 
that recommends extending the JSF development schedule to allow for further 


8 



65 


maturation of technologies. Their recommendation is based on their understanding 
that critical technologies will have inadequate levels of technology maturity, based 
on TRLs, at the time the EMD contract is to be awarded in Spring 2001. 

The GAO report clearly articulates a strategy for systems development that 
we embrace, as evidenced by the revisions to the DoD 5000 I mentioned earlier. 
Indeed, for the most part, we are in violent agreement. Where we differ is on the 
definition and applicability of TRLs in the decision making process. GAO’s position 
appears to be that achieving a level of TRL 7 should be required prior to entering 
EMD. Our view, however, is that a TRL 7 requires the very kind of systems 
integration that takes place during EMD and that it is infeasible to produce full 
scale testing of this type prior to EMD. Moreover, as noted earlier, we, like NASA 
and others, see TRLs as but one input to the decision process, albeit an important 
one. And it is one that varies from program to program. In fact, as the NASA 
Director of Programs told us: 

“NASA does not formally use nor rigorously apply the definitions of technology 
readiness level (TRL) to its systems development. We generally commit to development 
of operational systems when all critical technologies have achieved TRLs of 5 or 6.” 

In assessing a TRL prior to EMD, we seek to assess whether the individual 
technology has been proven, or is close to being proven, in appropriate 
developmental environments. We do not agree that it is necessary, or even feasible, 
to demonstrate the full integration of the technology prior to EMD. Thus, if we take 
out the requirement to fully integrate the technologies prior to EMD, we believe that 
our TRLs on virtually all critical technologies will be at least at the 5 or 6 level 
NASA generally uses, but more likely at 6 & 7 at the time the JSF goes into EMD. 

Further, it is important to note that the GAO arrived at their 
recommendation based on TRLs that the JSF Program Office and contractors used 
to grade the technology. Their original rating was due to the rationale that 
technologies were not being flown on the actual JSF platform. However, not all 
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technologies must be flown to be demonstrated. Much of the JSF avionics and 
software, for example, can and should be demonstrated on the ground at a lower 
cost. Technologies that must he flown for adequate demonstration, will in fact be, 
or have been, flying on the concept demonstrator aircraft, commercial aircraft, 
F-lb, F-22, F/A-18E/F and the Eurofighter. So, in short, although the technology 
will not be demonstrated on the actual JSF, it will be tested in a relevant 
environment - flown on similar aircraft. And the success of those demonstrations is 
critical to our confidence in our ability to successfully integrate the technologies 
during EMD. Therefore, the Department agrees that TRLs are an important 
measure, but alone, are not sufficient to decide when and where to insert new 
technologies into a system. 

GAO also expressed concern that if we make a premature production 
decision, we could be locked into manufacturing processes based on an expected 
technology capability, thus creating the risk that if the integration fails, we will face 
exorbitant cost and time in redevelopment. We agree there is risk. And how the 
JSF Program mitigates those risks is the key. As I think you will hear from Major 
General Huot, the risk mitigation initiatives associated with JSF have probably been 
the most comprehensive and aggressive of any DoD program ever. Further, as I 
noted before, the rapid maturation of modeling and simulation capabilities has 
enabled the development and testing of a wide range of critical technologies that 
might not otherwise been possible and has played a key role in our ability to 
effectively assess the risks on the JSF program. Thus, any decision to proceed into 
production of the JSF, will carefully assess all risks against the fallback alternatives. 

Finally, let me be very clear. The strategy 1 have articulated for the revised 
acquisition process that will be prescribed in the DoD 5000 rewrite, and which is 
largely being implemented on the JSF, represents a real departure from our 
traditional approach to systems development. In the past, we would consider 
technology development as part of the engineering and manufacturing phase. Now, 
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we have a technology development phase before we make a commitment to proceed 
into system development and demonstration and, eventually, into production. 

Mr. Chairman, Acquisition Reform has, as I said before, been made possible 
by a strong partnership between the Congress and the Executive Branch. It reflects 
our mutual commitment to ensuring that the government operates far more 
efficiently and effectively than has historically been the case. I do appreciate having 
had this opportunity to appear before you today, because as we continue on the 
successful path we have forged, we need the continued commitment and support of 
Congress. We also believe that it is important to let programs like JSF, that are 
demonstrating the acquisition and management reforms of recent years, have the 
flexibility to manage their programs and make decisions based on weighing the risks 
against cost, schedule and technology maturity. 

I thank you and the committee for this opportunity to comment on 
Acquisition Reform in general, and as it relates to the JSF Program acquisition and 
development strategy. I hope the information I have provided will prove helpful. 

I am happy to answer any of your questions now and certainly stand ready to 
provide any additional information the committee believes would be helpful in 
fostering a clearer understanding of this and other DoD programs. 
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Mr. Shays. I don’t think I’ll be as generous with the third panel. 
We’ll stick with the 10 minutes. 

General, now you only have 2 minutes. 

Mr. Soloway. That’s the way it usually works between OSD and 
the services. 

Mr. Shays. General, you have your time. And we welcome your 
testimony. 

General Huot. Thank you, Mr. Chairman and members of the 
committee. I appreciate the opportunity to talk about the Joint 
Strike Fighter program and the significant accomplishments it has 
demonstrated in the area of acquisition reform. 

By way of background, for those of you on the committee who I 
haven’t had the opportunity to meet, I have been in the operational 
end of the warfighting business for most of my career. I flew the 
F-105 Thunderchief in combat in Southeast Asia during the Viet- 
nam conflict. Since then, I have flown several other attack aircraft, 
the A— 7, A— 10, F— 18, and almost every model of the F— 16. I was 
a wing commander during the Persian Gulf war. And since then, 
in my headquarters experience, I’ve been the Deputy Director of 
Operational Requirements and almost 2 years in this acquisition 
position as the mission area director responsible for fighters, bomb- 
ers and munitions, and the JSF program falls into that category. 

Now, the JSF program has been a leader in the implementation 
of acquisition reform since its inception as the Joint Advanced 
Strike Technology Program or the JAST program. You may recall 
that we combined Air Force and Navy requirements after we can- 
celled the multi-role fighter program and the AX, helped them 
meet joint service requirements. 

Now, the need to produce an affordable aircraft drove several key 
decisions early in the program, including a single engine, a single 
seat, and a common family of airplanes for all three services. Spe- 
cific advanced technologies were selected and then prioritized based 
on their contribution to not only warfighting benefits but also in 
terms of life cycle costs. 

We established aggressive cost goals for average unit recurring 
flyaway costs and for engineering and manufacturing development. 
And cost goals are being established even now for life cycle costs. 
These cost goals continue to serve as baseline independent vari- 
ables for requirements and technology affordability trades. And I’ll 
talk about that regard in a minute because this is a new process 
that we have gone through. It is a robust and highly successful cost 
and operational performance trade process has been implemented 
by the Joint Strike Fighter on a continuous basis in order to help 
achieve those cost goals. 

Participation by the three major stakeholders in this process — 
our warfighters, industry, and the government has been key to the 
success of this cost and operational performance trade process. The 
warfighters, represented by both operators and maintainers, con- 
tinue to provide a clear and unambiguous view on fighting concepts 
and needs as well as expected threats in combat conditions. 

And industry provides us with detailed weapons system concept 
and cost information and the program office adds an understanding 
of cost schedule performance supportability and resource con- 
straints. 
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More importantly, government engineers and analysts provide 
assessments on the cost and performance of contractor concepts. 
The result is that every requirement on this airplane has to earn 
its way onto the airplane based on cost effectiveness or cost-benefit 
analysis. 

Now, in contrast to some programs which are not initiated until 
a formal validated requirements document existed, the Joint Strike 
Fighter program was established so that specific weapons system 
requirements would not be frozen until the leveraging cost and 
operational performance trades had been performed and key tech- 
nologies and concepts had been matured. This just-in-time ap- 
proach to requirements avoided premature commitment to require- 
ments that would be excessively costly to meet, fail to take advan- 
tage of available technology or, conversely, depend upon immature 
technology. In other words, it allowed time to work down the cost 
of the weapons system and insures that the requirements are con- 
sistent with the available technology. 

I would like to spend just a few minutes discussing the current 
phase of the JSF Program Concept Demonstration Phase we’re in. 
This phase began in November 1996 and it’s scheduled to be com- 
plete in the spring of next year. The major activities during this 
phase are propulsion system development, requirements analysis 
and definition, the technology maturation programs and, of course, 
building and flying two concept demonstrator aircraft per contrac- 
tor. 

In regards to the concept demonstrator aircraft, I want to empha- 
size a couple points to the committee. These aircraft are concept 
demonstrator aircraft. They are not and were never intended to be 
prototypes. These demonstrator aircraft are required in the concept 
demonstration phase to accomplish three very specific objectives: 
One, to demonstrate a high degree of commonality across all three 
common service variants; to demonstrate short takeoff, the vertical 
landing, hover and transition to and from forward flight; and then 
demonstrate satisfactory low-speed carrier approach flying and 
handling qualities. 

In fact, there are other items to be demonstrated by the contrac- 
tors on these airplanes but these are for competitive advantage 
purposes. But the hard requirements are the three things I listed 
above. 

We are totally convinced the JSF program office and our contrac- 
tor teams have done a great job in ensuring that the concept devel- 
opment phase of this program will demonstrate a low level of tech- 
nology risk for critical enabling technology and processes prior to 
entering EMD in the early part of next year on the current sched- 
ule. 

This has been accomplished through a very rigorous and dis- 
ciplined methodology for risk assessment, risk management, and 
risk reduction. Now, Pratt & Whitney is developing engines for 
both the JSF contractors’ demonstrator aircraft based on their 
highly successful development program for the F-22. The result 
here is a high degree of commonality not only among the JSF con- 
tractors but also between the Joint Strike Fighter and the F-22. 
The F— 119 engine core is essentially the same for both the F-22 
and the Joint Strike Fighter. This commonality lowers JSF risk, 
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development time, life cycle cost, and accelerates that propulsion 
system maturity so we can provide a safer airplane for our 
warfighters. 

Under the current schedule, in fact, we expect to have over 
500,000 hours of operational F-22 engine time before we have that 
engine at initial operational capability in the JSF. This is more 
than double what we had on the F-100 engine in the F— 15 when 
we transitioned to the single engine F— 16. 

I would like to just talk a little bit about the last two concept 
demonstrator — concept development goals — technology maturation 
and concept demonstrator aircraft, a bit further. The JSF program 
identified numerous technology maturation efforts to ensure low 
risk into engineering and manufacturing development. Now these 
technology maturation efforts aim to fulfill two key recommenda- 
tions of the 1985-1986 Packard Commission: to apply advanced 
technology to reduce costs, not just to increase performance, and to 
demonstrate advanced technologies prior to the start of the EMD. 

In the Single Acquisition Management Plan [SAMP], that was 
signed by Dr. Kaminski in November 1996 it clearly states that the 
goal of “tech mat” program is to evolve the most promising leading- 
edge technologies to a low level of risk prior to integration into the 
JSF EMD program. 

It’s important to emphasize that integration and its correspond- 
ing risk is, and always was, to be addressed in EMD. Exit criteria 
from the concept development phase states the Joint Strike Fighter 
will demonstrate to a low level of technical risk those critical ena- 
bling technologies, processes, and system characteristics. Integra- 
tion on the JSF program is the focus of EMD. 

Now the Joint Strike Fighter Program Office, in conjunction with 
each competing contractor, identified critical technologies processes 
and system characteristics required for the program tailored to 
each contractor design. Robust risk management processes were es- 
tablished by each competing contractor and validated by the pro- 
gram office. The government did not specify to the contractors 
which techniques must be used to track risk. The contractors se- 
lected what they felt were the best methodologies to accomplish 
that task. 

Now the government has been provided real-time access to those 
systems, actually on a computer data base, for oversight and re- 
view during the entire phase of the program. Both contractors uti- 
lize what is known as “waterfall charts” using a Willoughby tem- 
plate. This is a common and accepted methodology in industry and 
government. In fact, it’s taught at our Defense Systems Manage- 
ment College where risks have been identified, baselined, and 
tracked to document the very specific events required to reduce the 
risk of these critical technologies, processes, and system character- 
istics to a low level of risk prior to EMD initiation. 

Implementation of that risk management strategy has not 
changed since the program entered the concept development phase 
in 1996. And, most significantly, all of the critical technologies have 
achieved or are on track to achieve a low level of risk prior to the 
start of EMD. 

I want to assure the committee that DOD is convinced that our 
JSF weapons system contractors are appropriately reducing the 
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risk of these technologies through an affordable mix of flight and 
ground demonstrations, component demonstrations, and modeling 
and simulation and analysis. Based on a request from the Sub- 
committee on Military Procurement of the House Armed Services 
Committee, both our JSF prime contractors and Pratt & Whitney 
recently reaffirmed in detailed written responses, which were also 
shared with the GAO, that sufficient testing and demonstration is 
in place for the Joint Strike Fighter program to enter EMD at low 
risk. 

In summary, clearly since the program’s inception almost 6 years 
ago, the program office has been following a rigorous risk reduction 
plan. The risk reduction plan is on track to reduce the risk of each 
technology to low, by entry into EMD, and leave the integration of 
those technologies to the EMD phase where it belongs. This risk re- 
duction effort has been an important part in the program’s overall 
goals to implement acquisition reform. 

This Joint Strike Fighter is vital to the modernization of all of 
our services’ air forces and many of our closest allies. Any signifi- 
cant delay to the program would result in increased costs and also 
have serious impacts on our force structure and readiness. This 
was highlighted in detail in a recent May 2 Deputy Secretary of 
Defense letter to our services’ most senior leadership. That letter 
was addressed to each of the service Secretaries, to the Chief of 
Staff of the Air Force, the Commandant of the Marine Corps, and 
the Chief of Naval Operations. 

The JSF was chartered to do business differently to demonstrate 
leadership in acquisition reform, and it has done this. Having em- 
braced these concepts, it was rewarding to those who have worked 
so hard on this program to be presented the DOD David Packard 
Excellence in Acquisition Award in March 1997. All of this has 
been accomplished under the twin goals of developing an affordable 
weapons system that can meet the warfighters’ needs well into the 
21st century while helping to reform the acquisition process. 

So, Mr. Chairman, when you ask, “Joint Strike Fighter, acquisi- 
tion reform: Will it fly?”, my answer to you is that JSF has already 
demonstrated acquisition reforms as called for by Congress, and it 
will continue to write the book that future programs will follow. 
Thank you. 

Mr. Shays. Thank you. 

[The prepared statement of General Huot follows:] 
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Mr. Chairman and members of the Committee, thank you for this opportunity to appear 
before you to discuss the Joint Strike Fighter and the accomplishments it has demonstrated in the 
area of acquisition reform. The most important part of the vision has been to develop and 
produce an affordable next-generation strike fighter which meets the needs of the warfighters 
well into the 21 st century. The other part of the vision of the JSF Program has always been to be 
the model acquisition program for joint service and international cooperation. 

IMPLEMENTATION OF ACQUISITION REFORM 

The Joint Strike Fighter (JSF) program has been a leader in the implementation of 
acquisition reform since its inception as the Joint Advanced Strike Technology (JAST) Program 
in late 1993. Since then, the program pioneered the application of many acquisition reform 
concepts and has been awarded numerous awards for its efforts. 

One of the most important features of the JSF Program organization is the extent to 
which it is truly a joint program. This has been achieved largely through careful planning as set 
forth in the original JAST Charter. A significant arrangement set forth in the charter is that no 
single executive service for the JSF Program exists. Rather, the directorship rotates every two to 
three years between the Department of the Air Force and the Department of the Navy. 
Additionally, the Program Director reports through the Service Acquisition Executive of the 
opposite department. Such an arrangement has fostered the environment where both services are 
absolutely convinced they own this program. 

As the precursor to the Joint Strike Fighter, JAST was originally chartered as a 
comprehensive, advanced technology effort— a non-acquisition category (ACAT) program. 
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There were two main reasons for this. First, it was very specifically intended that the program 
should not focus prematurely on the development of any specific weapons systems, but rather 
should allow such decisions to fall out naturally from the interaction between the requirements 
and the technology communities. The second reason was to facilitate the implementation of 
streamlined and innovative processes. 

The non- AC AT status meant the JAST Program Director could use policies and 
procedures that w'ould facilitate the accomplishment of the program's charter to provide 
significant cost savings, and/or lead the way in demonstrating new processes that may eventually 
be incorporated into the mainstream acquisition system. Proper controls and checks and 
balances were utilized and the JAST program Director’s reporting channels were similar to those 
of a major, high-visibility acquisition program. The program was still reviewed at the same 
levels and by many of the same groups that would review an AC AT 1 D program. As a result, 
when JAST became JSF and was designated an ACAT ID acquisition program in early 1996, no 
dramatic changes in the way the program was managed were required. The program was already 
reporting through the same channels as an ACAT ID program. 

The JSF program has implemented many initiatives to streamline its business operations 
and to facilitate the development and procurement of the best possible products at the lowest 
possible cost. A few of the new business practices implemented include an Acquisition Reform 
and Streamlining Focus Team, electronic source selection, and extensive cost and performance 
trades in validating the “’operational” requirement. 
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FOCUS ON AFFORDABILITY 


As mentioned earlier, part of the vision for JSF is to produce an affordable weapon 
system that meets the warfighters' needs. To accomplish this, JSF has been a flagship program 
for the implementation of Cost as an Independent Variable or CAIV. As a flagship, JSF is not 
only practicing CAIV. but also developing techniques and applications and cataloging lessons 
learned for the benefit of other programs. Additionally. JSF has not only concentrated on 
acquisition cost, but also on sustainment costs. A Life Cycle Cost model was adopted during 
the first few months of the program and continues to be refined. This is probably the most 
mature life cycle cost model ever to be in place in a program at such an early stage. 

Affordability drove several key decisions early in the program — including a single 
engine, a single seat, and the highly common "tri-service family of aircraft 1 ' concept. Specific 
advance technologies were then prioritized based on their contribution to life cycle cost savings 
and warfighting benefits. These technologies will be the focus of later discussion. Finally, cost 
goals were established for average unit recurring flyaway cost and EMD cost, and are being 
established for life cycle costs. These serve as baseline independent variables for requirements 
and technology affordability trades. 

The Cost and Operational Performance Trades (COPT) process has been implemented by 
JSF on a continuous basis. Ensuring participation by the three major stakeholders in a weapon 
system development-warfighters, industry, and government, has been the key to the success of 
this process. Warfighters represented by both operators and maintainers contribute a clear and 
unambiguous focus on warfighting concepts and needs, as well as expected combat conditions. 
They are supported by representatives from the intelligence community which contributes 
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information on the threat environment in which the weapon system will exist. Furthermore, the 
emerging Command, Control, Communications, and Computers/Intelligence, Surveillance, and 
Reconnaissance (C4ISR) community defines linkages to C4ISR infrastructure. Finally, the 
modeling and simulation community provides digital representations of weapon and support 
concepts early in development to support multiple cost performance trades. 

Industry provides detailed weapon system concept and cost information that makes the 
cost-performance trade process function. The program office adds an understanding of cost, 
schedule, performance, supportability, and resource constraints. Government engineers and 
analysts provide analyses and technology explorations outside the scope of industry’s detailed 
concept developments. More importantly, government engineers and analysts present 
assessments on the cost and performance of contractor concepts. 

The result is that every requirement earns its way onto the aircraft on the basis of cost 
effectiveness. The COPT process was coordinated with the development of each Joint Initial 
Requirements Document (JIRD) update. This link promoted iterative and interactive 
requirements and cost trades, w'hich culminated in a validated Operational Requirements 
Document. 

In contrast to some programs which were not initiated until a formal, validated 
requirements document existed, JAST was established to meet a set of broadly defined needs that 
were acknowledged in the Bottom Up Review. This directed that specific weapon system 
requirements should not be frozen until the leveraging cost/performance trades had been 
performed, and key technologies and concepts had been matured. The COPT and JIRD 
processes supported this concept. This "just in time" approach avoided premature commitment 
to requirements that are excessively costly to meet, fail to take advantage of available 
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technology, or conversely depend upon immature or overly expensive technology. In other 
words, it allows time to work down the cost of the weapons system and insures that the 
requirements are consistent with the available technology. I would like to concentrate now on 
the current phase of the JSF program — the Concept Demonstration Phase. 


GOALS OF CURRENT PHASE: 
CONCEPT DEMONSTRATION PHASE 


The Concept Demonstration Phase began 16 Nov 1996 with the award of contracts to the 
Boeing Company and the Lockheed Martin Corporation. This phase is scheduled to reach 
completion in the spring of next year. The major activities during CDP are propulsion system 
development, requirements analysis and definition, technology maturation programs, and of 
course, build and fly two concept demonstrator aircraft per contractor. There are a few important 
points to consider regarding the concept demonstrator aircraft. These aircraft are “concept 
demonstrator” aircraft. They are not and were never intended to be prototypes. However, they 
were put in the Concept Demonstration Phase to accomplish three, very specific objectives: 1 .) 
demonstrate a high degree of commonality and modularity among the services three JSF 
variants, 2.) demonstrate short takeoff, vertical landing (STOVL), hover, and transition to and 
from forward flight, and 3.) demonstrate satisfactory low speed aircraft carrier approach flying 
and handling qualities. 
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TECHNOLOGIES LOW RISK AT ENTRY TO EMD 


The JSF program office has done an admirable job in ensuring that the CDP program will 
deliver capability at a low level of risk in order to begin integrating the functionalities during 
EMD. 

Pratt and Whitney is developing engines for both prime JSF contractors' demonstrator 
aircraft based on their successful engine development program for the F-22. The result is a high 
degree of commonality not only between JSF contractors, but also between JSF and F-22. This 
commonality lowers JSF risk, development time, life cycle costs and accelerates propulsion 
system maturity' for increased single engine safety. The bottom line is that all F-22 F-l 19 engine 
lessons learned have been incorporated and these engines are on track and will begin flying this 
summer. 

The requirements analysis and definition portion of this phase recently reached a 
successful conclusion with ORD approval by the JROC. This process actually began with JIRD 
I in Nov 1995. This 5 year process accomplished what is was asked to accomplish — the most 
cost effective solution which meets the w'arfighters' needs. This was done by continually trading 
cost and performance with each successive JIRD process, tightening the trade space available as 
a solution was homed in on. 

I will discuss the last two CDP goals — technology maturation and concept demonstrator 
aircraft— as they counter the GAO's claim that JSF will not be mature enough to enter EMD. 

The JAST and later JSF programs had many technology maturation efforts to ensure the 
very thing the GAO claimed — high risk entry into EMD — would not be an issue. These 
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technology maturation efforts aim to fulfill two key recommendations of the 1985-86 Packard 
Commission: 1) apply advanced technology to reduce cost, not just to increase performance; and 
2) demonstrate advanced technologies prior to the start of EMD. The plan was laid out by the 
former Under Secretary of Defense, Dr Kaminski in 1994. 

The Joint Strike Fighter Program Office, in conjunction with each competing contractor, 
identified critical technologies, processes, and system characteristics required for the program, 
tailored to each contractor’s design. Robust risk management processes were established by each 
competing contractor and validated by the program office. The government is provided real-time 
access to these systems for oversight and review during this entire phase of the program. Both 
contractors utilize w'hat is known as “waterfall charts’' built using a Willoughby template. This 
is a common and accepted practice in industry and government. Risk waterfalls enable joint 
reconciliation of risk level between the government and each contractor at any given time. Risks 
have been identified, baselined, and tracked to document the specific events required to reduce 
the risk of these critical technologies, processes, and system characteristics to a low level prior to 
EMD initiation. Implementation of this acquisition strategy and risk management strategy has 
not changed since the program entered CDP in 1996 and most significantly, all of the critical 
technologies areas have achieved or are on track to reach a low level of risk prior to the start of 
EMD. 

The GAO report highlighted eight critical technologies. These are some of the 
technologies that are part of the technology maturation program. The heart of the GAO concern 
is based on a risk assessment tool, which the JSF program office does not use, applied to these 
eight technologies. Due to the proprietary nature of the program I cannot discuss the details of 
each one of these technology areas. However, I would like to assure the Committee that DoD is 
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convinced both weapons system contractors are appropriately reducing the risks of these 
technologies through an affordable mix of flight and ground demonstrations, component 
demonstrations, and modeling, simulation and analysis. Additionally at the request of the House 
Armed Services Committee, both weapons system contractors were asked to provide their 
assessment of the technology risk in these 8 areas, and the program overall. Both reported that 
they feel the program will enter EMD at low risk, and that sufficient testing and demonstration is 
in place in the CDP phase to ensure low technology risk entry into EMD. 


SUMMARY 


Clearly since the program’s inception, the program office has been following a rigorous risk 
reduction plan for approximately 6 years. The risk reduction plan is on track to reduce risk of 
each technology to "low" by the entry into EMD and leave the integration of these technologies 
to the EMD phase where it belongs. This risk reduction effort has been an important part of the 
program's overall goals to implement acquisition reform. In addition, the program's focus on 
affordability resulted in cost and operational performance trades which delivered a capable, 
affordable weapons system. The JSF is vital to the modernization of not only the United States 
but also our closest allies. Any unnecessary delay to the program, such as the one being 
suggested by the GAO, would not only impact the cost of the program but also have a serious 
impact on our force structure. 

The JSF/JAST program was chartered to do business differently and to demonstrate 
leadership in acquisition reform, and it has done this. This can be seen from how the program is 
organized to the goals laid out for the Concept Development Phase. Having embraced these 


VERSION 8 



81 


concepts, it was rewarding to those who have worked hard to make this a reality to be presented 
the DOD David Packard Excellence in Acquisition Award in March 1997. 

All this has been accomplished under the twin goals of developing an affordable weapons 
system that can meet the warfighters' needs well into the 21 st century while reforming the 
acquisition process. So Mr. Chairman, when you ask "JSF Acquisition Reform: Will It Fly?", 
my answer to you is JSF has already demonstrated reforms as called for by Congress and it will 
continue to write the book that future programs will follow. 
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Mr. Shays. It’s great to have you both here and your statements 
were helpful. And I would like to ask a number of questions, stat- 
ing first that I think we on both sides of the table agree that JSF 
is a necessary program. We’re going to see this plane built and all 
its variations. And so it’s just really a question of how to proceed. 
And — but I do think there are some substantive differences be- 
tween GAO and your position, DOD’s position. And I’d like to in- 
vestigate that a little bit. But I would also like to acknowledge, you 
know, where we may agree, so then we can just really focus on 
where we need to focus on. 

The bottom line is that the new attack fighter, the AFX for the 
Navy and the multi-role fighter, MRF, for the Air Force, is dropped 
and we don’t see any hope or need to resurrect that. That’s pretty 
much off the table. So it makes JSF even more important. 

I think we both can agree — so you nodded your heads. I take that 
as a yes. You’re going to have to speak. Actually it’s very hard for 
her to take a nodding of a head. Since I didn’t address it to one, 
I understand. So I’ll start with you, Mr. Soloway. And General 
Huot, if you disagree, you certainly would step in. So I assume if 
Mr. Soloway says — gives the answer, you are in agreement with 
him and vice versa. Is that fair enough? 

Mr. Soloway. OK. 

Mr. Shays. So the AFX and MRF are off the table and we agree 
that JSF, Joint Strike Fighter, is our next plane, in addition to the 
F-22 and the F/A-18E and F. Would you agree with GAO that the 
F/A-18E and F is a significantly different plane, almost a brand 
new plane, or would you contend that it is a variation on a plane 
that we already have? I’ll ask both of you. 

Mr. Soloway. Actually, the general has the expertise. 

General Huot. Mr. Chairman, I really don’t feel qualified to pro- 
vide a specific answer there. I don’t know that much about the spe- 
cific differences between those two aircraft. 

Mr. Shays. I’ve always known the nonmilitary side to be willing 
to give an answer on this. So, Mr. Soloway. 

Mr. Soloway. But I’ve already passed to the general. 

Mr. Shays. So that’s going to be — it’s going to stand basically, 
that comment, that it is a significantly different plane by the GAO, 
if it’s not refuted. 

Mr. Soloway. That would be my understanding, but we can cer- 
tainly get you more detail on what the differences are. 

General Huot. It may be appropriate, Mr. Chairman, to go to the 
Navy and answer that question. 

Mr. Shays. OK. Fair enough. We would agree that the Joint — 
JSF is intended to do three basic tasks: for the Air Force, a conven- 
tional airplane, tactical airplane; for the Navy, slow-speed struc- 
ture that has to be a little more durable to take the harder land- 
ings and the shorter landings; and for the Marines, a vertical take- 
off. And those are, General — I make an assumption — three very dif- 
ferent tasks. 

General Huot. Yes, Mr. Chairman. Probably the best way to 
clarify that, for the Air Force the Joint Strike Fighter will replace 
F-16s and A-lO’s. For the Marine Corps they will replace the AV- 
8s and some of their F-18s. And for the Navy they will complement 
their F-18E/F force. And in the three variants of the airplane we 
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achieve — we plan to achieve a high degree of commonality, and 
right now that appears to be somewhere between 70 and 80 per- 
cent commonality between each of the three variants. There is the 
conventional takeoff-and-landing variant for the Air Force. There is 
the carrier-suited variant for the Navy, and the STOVL variant, 
the short-takeoff-and-vertical-landing variant, that will satisfy the 
needs not only for our Marine Corps, but also for some of our for- 
eign participants, notably the United Kingdom. 

Mr. Shays. Things like the avionics would be similar in all 
three? 

General Huot. Right now the avionics are planned to be common 
in all three variants. 

Mr. Shays. So some will be the same? 

General Huot. Significant savings there in terms of commonal- 
ity. 

Mr. Shays. But admittedly we tried this in the past where we 
tried to have one plane meet the needs of more than one branch — 
Navy, Marines, Air Force — and we haven’t always had — it hasn’t 
been all that successful, correct? I know we have 

General Huot. I would say that the F-4 is probably an example 
that you might refer to where it didn’t meet everyone’s require- 
ments. 

I would point to the fact in this program and as a warfighter I 
would say this has been a great process. This Cost and Operational 
Performance Trades process that I talked about in my oral testi- 
mony, this is where we had what we call an ops advisory group, 
an OAG, a group of warfighters that got together and worked with 
the contractor and the SPO, the systems program office; and we 
worked this requirements process to evolve the requirements for 
the services over a 5-year period. And in fact we finally got their 
final Operational Requirements Document approved by the Joint 
Requirements Oversight Council, that’s a group chaired by the vice 
chairman of the Joint Chiefs and with membership of the vice 
chiefs of each of the services, approved on March 13th. 

And as I said, what we did in there is, we went carefully through 
to make sure that each of the services gets the requirements that 
they need to do their individual job. So, you see, in each of these 
variants there will be some unique requirements to meet unique 
service requirements, but at the same time achieve the greatest de- 
gree of commonality that we can. And we think 

Mr. Shays. But the bottom line is — and I accept what you’re say- 
ing, but the F-4 was an example of a plane that didn’t do quite 
enough for any of the branches. It’s not to say we shouldn’t do it 
again to try to make it work, but this is a significant undertaking. 
Because it is not just for the Navy and it’s not just for the Air 
Force, it’s not just for the Marines; it’s for all three with a variation 
of basically between 20 and 30 percent from one to the other. 

And I would like to know if you agree or disagree with GAO 
when they said that if the technology development is $1, if they 
haven’t developed the technology by production development — prod- 
uct development, excuse me — that it’s $10, and that if they’ve got- 
ten to the point of production, that $1 becomes $100. 

General Huot. Mr. Chairman I have never heard those numbers 
before. 
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Mr. Shays. So you would want us to substantiate those num- 
bers? 

General Huot. I would have no way of commenting on those. 

Mr. Shays. Fine. If you would speak just a little louder. But the 
bottom line is you have no way. Fair enough. 

Intuitively, to me, that is not an unrealistic number. In other 
words I wouldn’t be surprised if that were the case, but I accept 
for the time being that it hasn’t been documented. But it is now 
part of the record as a statement that we need to address. 

Mr. Soloway. I think that we can — we will take the question for 
the record also, but I am not aware of any program that we have 
ever done that has had a 100 fold increase in cost from 

Mr. Shays. But it may just be that aspect. 

Mr. Soloway. Or even that aspect, I’m just not aware of it. 

Mr. Shays. Let me clarify. Not that the whole program goes up, 
but that particular failure of technology has resulted in a 10 to 1 
in product development and a 100 to 1 in production. So on the 
table — and your point is you’re not aware of that kind of — but intu- 
itively we would agree that costs would go up significantly, and we 
have a heck of a lot of past history to document that that’s the 
case. 

Now, I see a difference between the logic when you’re building 
a school building your technology is there, but you haven’t designed 
it. So you say, let’s do it all at once, and in many cases it’s proved 
to be cheaper and you get the product sooner. You don’t wait to de- 
sign everything; some things happen in process. But this is a bit 
different, correct? I can’t use that same analogy, and the DOD 
would be — wouldn’t use that same analogy. 

Mr. Soloway. What we would say, sir, on that question is that 
we are in fact demonstrating the critical-path technologies prior to 
going into EMD. The difference really comes down to how one de- 
fines technology readiness levels as a means of measuring where 
you are in that process. We are not entering into EMD where we 
have critical-path technologies that either don’t exist or we haven’t 
demonstrated in what we call a relevant environment. But the dif- 
ference really comes down to — and if you look, for instance, at the 
way in which the technology readiness levels are defined, even in 
the GAO report where at level 6 they talk in terms of system or 
subsystem models or prototypes demonstration in a relevant envi- 
ronment, we will have done that, I believe, in every technology, 
every critical-path technology prior to EMD. Level 7 speaks to a 
system prototype. 

Mr. Shays. Could someone put up that chart that GAO had. It 
was figure 4 in their report. That’s the technologies 1 through 8. 

Mr. Soloway. My only point was going to be 

Mr. Shays. I know, but you were making reference to it, so let’s 
just leave it. Thank you very much. 

Mr. Soloway. When you talk in terms of technology readiness 
level 7, which Mr. Rodrigues said they believe should be the 
threshold test before you go into engineering and manufacturing 
development, our view on that is that definition of TRL 7 speaks 
to a system prototype demonstration in an operational environ- 
ment. That suggests to us that you have had to do a great deal 
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about both systems, engineering and systems integration, because 
you’re dealing here with a system of systems. 

Mr. Shays. What does it mean, though, when they say in a “test 
bed aircraft.” 

Mr. Soloway. A test bed aircraft can be any aircraft you’re flying 
to test a given — it might not be one aircraft which — I said in my 
testimony, we have the CDA, we’re using commercial aircraft, the 
F-16s, F-18s, F-22, Eurofighter; various different technologies 
that need to fly will fly in an aircraft 

Mr. Shays. But it isn’t a prototype then? 

Mr. Soloway. No, it’s not a prototype, though; that’s correct. 
That’s the key difference. 

Mr. Shays. The question that my counsel is asking is, who said 
it was? I mean, are you putting up a strong 

Mr. Soloway. I’m sorry? 

Mr. Shays. That it was a prototype? 

Mr. Soloway. It says under the definition of “technology readi- 
ness levels” in the GAO report of last July the definition of a level 
7 is a system prototype demonstration, and I think if — without try- 
ing to get 

Mr. Shays. Then it says, examples include testing the prototype. 

Mr. Soloway. Testing the prototype. 

Mr. Shays. So we have eight basic technologies and they don’t 
technically have to be tested on a prototype plane; they can be test- 
ed on another plane, correct? They could — so 

Mr. Soloway. Well, you get into various discussions. For in- 
stance, on some of the — we can’t get into specific 

Mr. Shays. I think this is a key point. 

Mr. Soloway. It is a very key point, I agree. 

Mr. Shays. And it would be wrong to suggest that somehow you 
have to have a prototype. The question is, can you test it in other 
ways to know it works before we move forward with development? 

Mr. Soloway. Sir, actually — and I’m going to give you how I 
read their chart, and then General Huot may have a different sort 
of interpretation. 

I think there’s actually a great deal of grey area about the TRLs. 
They’re not quite as clear as has been defined, and I think that’s 
one of the reasons NASA and others have said you have to have 
some flexibility. 

Let me use the language out of the report that says, examples 
include testing the prototype in a test bed aircraft. So you have to 
have a prototype of the system. Now, does that mean in full form, 
fit and function where, for instance, the radar array has to be the 
full size, full scale that you’re going to use; or if technology allows 
you to, would you be testing a variant of the radar where the basic 
technology exists on a — for instance, an F-18 or some other air- 
craft. 

Others would be in the avionics. As I noted and I believe General 
Huot has talked to, the avionics don’t necessarily need to be flown 
on a test bed aircraft. They can be tested in an environment which 
is tremendously — has great integrity with regard to the operational 
requirements. 
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So it’s not an automatic, in our view, threshold that says you 
have to test everything in a prototype which involves, essentially 
full scale, for each individual technology? 

General Huot. Mr. Chairman, could I make just 

Mr. Shays. Let me just ask this point. The laboratory goes from 
one to three, correct, and then it gets out of the laboratory? 

General Huot. Those definitions become fuzzy in the sense that 
in order to demonstrate some technologies on a low level, you don’t 
necessarily have to fly them at all. You may be able to do that in 
an environment that never involves putting that technology in an 
airplane. And so what the program did — let me talk about what the 
program did a little bit. 

The program did not use technology readiness levels, primarily 
because they didn’t include a risk management process. What they 
did do is, they used these waterfall charts, or Willoughby templates 
where they identified all of the key technologies that needed to be 
reduced to low risk before going into EMD. 

Now, these — this process identifies the technology and then lays 
out over a schedule the critical events that have to happen to re- 
duce that specific technology to a low level of readiness. In addition 
to that, it provides off ramps, if you will, where if that technology 
is not going to work, they would revert to an alternate technology. 

This was chosen by the contractors as a way of not only assess- 
ing risk, which the TRL process does, but also managing risk and 
being sure that you get to that low level of risk when you’re ready 
to go into EMD. And that’s the process that they used and that’s 
the process that the program office monitored each of the contrac- 
tors on a 

Mr. Shays. Where you both agree, and then we’ll define it: You 
both agree you shouldn’t go into EMD before those critical tech- 
nology — before you come to low-level risk. 

General Huot. Exactly. 

Mr. Shays. So the debate now is whether what you described as 
low-level risk is low-level risk. I mean, that’s going to be — because 
you buy into the point, though, that you don’t move forward if 
you’re at high-level risk. And high-level risk is where you don’t 
have the technology. 

Mr. Soloway. There’s always — as I said in my testimony, when 
you’re dealing particularly in this kind of complexity of systems, 
the risk of integration, which is what you do in EMD, is always 
there. I mean, to say that we can whittle that risk down to a very 
low level in this level of complexity is not realistic. What we are 
looking to do, just to make sure we’re clear on the terminology, is 
to arrive at a very low level of risk relative to the individual tech- 
nologies. 

Mr. Shays. Let me just tell you what makes me nervous. You 
have described — you basically have described the technical readi- 
ness levels as tools rather than requirements. And then you’re 
doing one thing more; as I hear it, you’re also in a sense redefining 
low-level risk. And I thought at least we could get to the point 
where we agree you don’t move forward unless you’re at low-level 
risk. 

You’re trying to qualify low-level risk in a way that makes me 
uncomfortable. 
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Mr. Soloway. I just don’t want to leave on the table a suggestion 
that when you go to the integration and EMD that your risks are 
automatically low. What you have done is substantially reduce 
your risk by proving the individual technologies. But you still have 
a high-risk element because this is a complex undertaking. 

Mr. Shays. Well, I’m just going to tell you, that leaves me a little 
uneasy because that allows you to define it in a way that allows 
to you move forward, no matter, as I hear it. 

Mr. Soloway. In the past, we would not necessarily in all of our 
systems development bring individual technologies to the level of 
maturity that the JSF is doing, that we will now be requiring in 
the new 5000 rewrite; that’s the individual technology levels. To 
get further than that into and integrate the system is what you do 
in engineering and manufacturing development. That is always — 
there is always risk there, as I said in my testimony. 

We have reduced that risk by proving out the individual tech- 
nologies. That’s the big step forward that we have taken over pre- 
vious practice. 

Mr. Shays. I feel like I’m dealing with a moving target here. I 
know you’re trying to be helpful and you’re trying to be responsive. 
What I want to say, and then we’ll just have our disagreements — 
what I want to say to you is, you view technical readiness levels 
as a tool. I view them as something more than a tool. I view them 
as more a requirement, and you view them more as a tool. 

I view low-level risk as — obviously, when you’re dealing with 
military hardware, there are risks, but I think that low-level and 
high-level are pretty clear. And then there are going to be different 
levels of low risk. 

Now, where I may have some question is, if you’ve leveled — if so 
much of one technology, say the structures and materials, you’ve 
got almost everything licked, but one thing, and that one thing is 
still there, but you don’t want to hold up the rest of the project be- 
cause everything else is ready, I would think that you may still 
have something that needs to be resolved; and you wouldn’t want 
to wait another year for the whole project if you want to start to 
integrate these technologies together. 

So I may be a little off with that and I’ll, you know, ask GAO 
to help me out on this later as well, and other witnesses. All I’m 
saying to you is, I’m uncomfortable by your referring to this as a 
tool and your redefining low-level risk. That’s where I’m uncomfort- 
able. 

Let me do this: I’m going to come back. I’m going to let counsel 
and minority staff ask questions as well, because in the process of 
their asking questions it helps to define what I want to continue 
to ask. 

Mr. Halloran. 

Mr. Halloran. You want him to go first? 

Mr. Rapallo. Sure. 

Mr. Shays. This is David Rapallo. 

Mr. Rapallo. Could you just describe what effects you mentioned 
briefly in your statements — what effects a delay would have, a 
delay of say 6 months? 

General Huot. I wrote down some specific notes to talk to that. 
I think probably the best source of information there now — there 
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was an attachment to Mr. DeLeon’s letter to our senior service 
leadership that talked about the impacts of delays. 

Obviously, if you delay the program any period of time, you are 
going to incur some costs of — increased costs in the program itself. 
But more importantly, in this particular case, you’re looking at a 
delay in development that could roll into a delay in production that 
could be more than 6 months. And if you will go back and look at 
the history of programs, some delays in development of a program 
have led to just that. 

If you were to incur any kind of a significant delay here, we 
would have some significant force structure impacts and some im- 
pacts on our readiness. The DEPSECDEF letter attachment looked 
at a scenario that went out as much as 3 years and said, if you 
delayed the program procurement phase by 3 years, that we would 
end up with about a three fighter wing equivalent force structure 
shortfall that we’d have to deal with in a couple of different ways. 
You could buy gap-filler aircraft, F-16s, more F-16s, or you could 
do service life extensions to those aircraft if that were possible. 

Now, we have done service life extensions to F-16s, so in gen- 
eral, we think that’s probably a “doable” do. How much you would 
have to do would have to be determined. 

In the case of the Marines, though, it appears that they would 
not have that kind of an option where there really isn’t felt to be, 
in my view or my understanding, a viable way to carry the AV- 
8 further into the future. So they would end up with a force struc- 
ture deficit in order to meet their military commitments worldwide. 

And without getting into the details of what that would mean, 
in terms of their ability to do their mission, that attachment in the 
DEPSECDEF letter talked about some of those impacts to our 
CINCs worldwide. So we think in the long run there are some very 
significant impacts if you would slip this in terms of force structure 
and readiness impacts. 

Mr. Soloway. I might add one other to that in terms of cost. The 
longer we delay a decision — and I don’t want to suggest that the 
decision — Mr. Chairman, you asked the question, one of the mem- 
bers asked the question about what is sacrosanct about a certain 
timeframe for a decision. 

We set out a timeframe in which we thought we could make a 
decision, and we still believe that within a year we’re going to be 
able to make an EMD decision. But it is driven by our requirement 
of having the technology maturity of where we think it needs to be 
to make that decision, not by any arbitrary line in the sand. 

But presuming for a moment that we have what we believe are 
adequate levels of technology maturity, you have to realize that we 
would be delaying that decision for a longer period of time. Carry- 
ing both contractors, for instance, which is going to continue costs 
and so forth, would have an impact on our international partners 
who are participating in the program and the potential of either 
pullout or other reopeners, if you will, in terms of the international 
relationship. 

So there are a number of sort of follow-on impacts that could 
transpire as well. 

Mr. Shays. I want Mr. Rapallo to continue to develop this line, 
but I just wanted to ask you the question again: You say “adequate 
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levels of technology and maturity.” That’s just a different term 
than “low level,” and “adequate” leaves me really 

Mr. Soloway. I apologize if I seem to be obfuscating, because I 
really wasn’t, sir. 

When you made the comment that we would be at low risk, we 
will have low risk relative to the individual technologies, yes, that 
is a requirement. And that is a requirement. What is not a require- 
ment necessarily is that TRLs be the measure of that risk. 

We are recommending TRLs, by the way, in the new 5000 just 
as a common language, but there are, as General Huot said, other 
ways in which one assesses, measures and manages to risk mitiga- 
tion and risk management other than TRLs. They themselves are 
not the only. 

Mr. Shays. I’m going to go back. You’ve made the point that the 
Harrier — I call it the Harrier jet; is that the British term for it? 
But the vertical takeoff jet is deteriorating significantly. It needs 
to be replaced soon. The question mark is, do you speed up every- 
thing because of that? And one of the questions — I have total ac- 
ceptance that if you’re in production, and then we in Congress do 
what we sometimes do: We say, instead of 30 planes, it’s 20 planes; 
we add costs, I understand that. I understand how we slow things 
in production, but frankly sometimes the technology isn’t there or 
is being revised. That’s one of the costs of slowing it down. 

But I have a harder time understanding how slowing down the 
decision before you go into development and production adds sig- 
nificantly to costs. And I’ll just have you come to that question 
afterwards, but you had mentioned it and I just want to — I would 
love you to continue. 

Mr. Rapallo. That would be — my only followup is, GAO has sort 
of stated or described a process where if you delay the EMD proc- 
ess, you can develop the technologies and ultimately save money 
and save time. Do you disagree with that? 

Mr. Soloway. No, I don’t disagree with that, but on this given 
program, I think what we’re saying is, we would not go into EMD 
if we did not have the technologies developed, so that the delay in 
time is not necessarily going to help. We believe the technologies 
will be at a level that allows us to make a responsible decision to 
move to EMD. 

Now, whether we agree that, based on a TRL chart, if one were 
to use that measure, one would need to be at TRL 7 or 6, that’s — 
that is actually the crux of the debate, and what does it require to 
achieve a TRL 7 under the construct. But clearly if the technologies 
are not there, you cannot — particularly the critical-path tech- 
nologies that we’re talking about, then you wouldn’t make that de- 
cision. 

So in this case, all we’re saying is, to say today that we ought 
to delay the program, when we believe the technologies are either 
there or close to there and will be there in the timeframe, there 
would be no reason to delay. Otherwise, if the technologies aren’t 
there, it’s correct, you would not want to go into EMD; you would 
want to take advantage of further development time. 

Mr. Rapallo. Thank you. 

Mr. Shays. If you want to come back, we will. 

Mr. Halloran. 
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Mr. Halloran. Well, I guess the followup that occurs then is, 
you believe that technology — to be ready, how will you know? If not 
with some more quantitative or objective measure than TRLs, how 
will you know that, say, the vertical-takeoff-and-landing tech- 
nology — for example, speaking hypothetically, how will you know 
whether it’s ready or not? How will you measure that? 

Mr. Soloway. First of all, as General Huot said, that there are 
things some people call the Willoughby templates, and so there are 
other methodologies that are used. And you may, I think, hear 
from other witnesses in terms of probability, as well as objective 
measures and so forth. 

But the other reality is, and I don’t mean to be glib when I say 
this, it’s either going to be working or it’s not. It’s either — the capa- 
bility to do what you are talking about has either been dem- 
onstrated or not demonstrated on a concept aircraft or what have 
you. The radar or whatever technology you’re talking about will 
have or will not have been demonstrated on another plane. The 
avionics will or will not have been demonstrated in a suitable lab- 
oratory environment. 

Actually, when the chairman mentioned laboratories at levels 1 
to 3, there is a category in level 6 that talks about a high-fidelity 
laboratory, different than sort of the low grade. So we will actually 
have demonstrated the individual technologies in a variety of dif- 
ferent ways most relevant to what we have to prove and see. 

General Huot. I think the short answer there is that we will 
continue this rigorous and disciplined approach that we have had 
using these waterfalls to track each one of these technologies and 
assure that it’s a low level of risk. In fact, in the input that we got 
from Boeing and Lockheed and Pratt & Whitney in the inputs re- 
quested by House Armed Services, the Subcommittee on Military 
Procurement, they went through, in great detail, each of the tech- 
nologies describing the process that they were using and reaffirm- 
ing that we would be at low level of risk prior to entry into EMD. 
That is really the tool that we have been using. And it is a risk 
management tool that tracks, is event based, and it also again al- 
lows you to make decisions if you need to go to an alternate tech- 
nology to be at low risk prior to EMD. 

Mr. Halloran. Stay with that for a second and describe the wa- 
terfall again. What I heard and I think what the chairman heard 
is that high risk plus risk mitigation plan equals low risk. 

General Huot. No, not a plan. You have to execute the plan and 
then you monitor the plan to assure you’ve gotten there. So the 

Mr. Halloran. That’s a quantitative way? 

General Huot. In a quantitative way. 

The program office goes through, and as the contractor completes 
each of the events that they’ve agreed are to track this from high 
risk to medium risk to low risk, they check them off. And so you 
end up essentially with a checklist that says you are at low risk 
in that technology area — you have done all the things that you said 
you needed to do to get there. 

Mr. Soloway. Let me just add one other thing, if I could, on that 
point. Once you’ve done that, you also have, and General Huot 
mentioned a couple of times, fallbacks. I think it’s important to un- 
derstand that as you go through this process, let’s say for instance 
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that we have a technology that was demonstrated on an F-16 and 
that we thought this is ready to go. For some reason, as we go for- 
ward, it doesn’t work. What do we have to have done prior to EMD 
to address that potentiality? That is where the fallback is. You 
have to have an assessment of what lower-level capabilities you 
can access without affecting the critical performance parameters of 
the plan. 

Mr. Halt, or an. And cost? 

Mr. Soloway. Certainly. 

Mr. Halloran. Because I was going there next anyway. I’m glad 
you raised that. Because GAO made the point that in a couple of 
these critical areas the fallback was quite expensive and did affect 
performance or the requirements anyway, for example, and yet you 
testified that the 5000 series calls for a fallback that it would be 
at a higher 

Mr. Soloway. The fallback itself, if it’s measured, a TRL would 
be higher, but it might not be as high technology. 

Mr. Halloran. But it involves a tradeoff of some kind. 

Mr. Soloway. There’s a tradeoff certainly, and you have to as- 
sess all that in this process. And you have critical performance pa- 
rameters that you have to meet and you can’t go below that. 

Mr. Halloran. Let me change the subject slightly and talk about 
the definition of threshold between technology maturation and inte- 
gration. 

It struck me that in rescoring the original contractors, TRL scor- 
ing, the Department kind of expanded the integration to get itself 
out from under some low scores here and to push things up to 6s 
and 7s, to where you could define your risks more clearly, or at 
least as lower. So talk about that threshold some more. 

I mean, integration, there’s an example in some testimony that 
comes in in a minute of a DOD system that looked fine in pieces, 
but when they came together they interfered with each other and 
didn’t work. And that’s clearly an integration issue, and it’s a sepa- 
rate set of risks, as you testify. But is getting to demonstration of 
form, fit and function in a proper environment, is that integration 
or is that maturization, maturity? 

Mr. Soloway. I would suggest that the way — if you’re taking 
that comment and weighing it against the way TRLs are laid out 
and what levels you hit, because it’s sort of the context of the ques- 
tion — that you have to do a fair amount of integration to do that. 
So my answer would be, yes, we think that — let me go back to your 
initial question, because what you really were asking is, did we try 
to jimmy numbers to get out from under a bad score. That was the 
first question. 

And my answer is, and I can say this as one who was not in- 
volved in that initial process, but have gone back over the last 
month or two and have met with all of the players, have had exten- 
sive discussions, services and so forth. I genuinely believe that 
there was, despite what Mr. Rodrigues said, a lack of understand- 
ing of how these were being applied, because what was being done 
was assessing levels relative to the risk of systems integration. 
This is for the JSF. And so when you do that, you are — your rel- 
ative risk is higher. 
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But when you’re weighing it against a question of, have you 
proven out the individual technology, which is what we believe and 
the 5000 really speaks to — and the new 5000 needs to be matured 
in a way we have not traditionally done before, before you go to 
EMD — when you do that, that is — the result is the rescored num- 
bers. 

Mr. Halloran. What was the source of the misunderstanding? 
As you say, the 5000 draft had been out a while. It had reached — 
it had received some resistance. 

Mr. Soloway. This was not relative to the 5000 rewrite. This 
was relative to the GAO 

Mr. Halloran. But the issue and the proposed role of TRLs, as 
you state in your testimony, at some point push would come to 
shove and a low TRL would prevent a technology from moving into 
EMD; that was pretty well known. 

What was the source of the misunderstanding? 

General Huot. How to grade TRLs? Our program office had 
never used TRLs. When they graded them to come up with these 
ratings that you see on the chart, they rated those, including the 
risk of integration on the Joint Strike Fighter. Of course, that’s 
going to give you a much higher risk; that’s why the lower ratings. 

They did not rate the individual technology area by itself without 
consideration of the risk of integration. When they went back and 
did that, in fact, that was spelled out in Dr. Schneiter’s letter that 
responded to the GAO report, he made it very clear that the De- 
partment position was that the Department does not agree with 
the conclusion, which is based on misinterpretation of a process for 
determining the readiness of technologies for incorporation in 
major systems; and went on to say later that the GAO ground rules 
for scoring technology rating levels included the risk of integrating 
the technology onto the JSF platform. 

The JSF program office used those ground rules to arrive at the 
ratings contained in the draft report, and upon review and discus- 
sion with other users of technology readiness levels, that program 
office determined that only the maturity of the technology, not its 
integration, should be rated to determine the readiness under 
EMD. 

One finds that technology risk is expected to be an acceptable 
level of EMD start. 

Mr. Halloran. Let’s go back to my original question. Define “in- 
tegration” — “integration” meaning form, fit and function down to 
the size it will have to perform next to other systems, or “integra- 
tion” meaning actually working with other systems on an airplane? 

If you define it that broadly, then of course you’ll, I think, dimin- 
ish the use effectiveness of TRLs as a threshold judgment to get 
into EMD. 

General Huot. Let me say just a couple things here. First of all, 
it was clear that some of these things have to fly on aircraft in 
order to get to the low level of risk. In fact, if you would bear with 
me just a moment if you look at those eight technology areas with- 
out addressing the eight technologies specifically — for example, 
technology No. 4 is flying on an F-16, technology No. 1 is flying 
on a concept demonstrator aircraft — in fact, a lot of these tech- 
nology areas will fly on aircraft. But when the program office made 
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their initial assessment using those, the TRL scoring, they included 
the risk of integration on the JSF platform to score those. 

Mr. Halloran. Let me finally, Mr. Soloway, in your testimony, 
in talking about the 5000 series, and you say — which is largely 
being implemented in JSF, speaking more broadly about acquisi- 
tion reform, now, what — in what way is it not largely — gives you 
some room out — in what way is it not — what challenges does the 
Department face in more fully applying — was this referring to this 
program and others that follow it? 

Mr. Soloway. You’ve got two separate questions. Let me start 
with the JSF specific and then move to the broader question if I 
could. 

On the JSF, when you — we complete the 5000 rewrite, which is 
now out for comment, we expect to have it finalized in the next 30 
to 60 days; it’s been a long process; it’s a fairly complex set of docu- 
ments — there will be actually a new systems acquisition model, if 
you will, with different kinds of entry and exit criteria based on 
much of the kind of model that GAO talked about this morning. 
And this is precisely where we are going in the Department, and 
the kinds of criteria that you have to meet to go into various 
phases of the process, and they may have different names, and it’s 
really a different model. 

JSF has, in effect, mirrored that model, but not precisely because 
the model didn’t even exist until the last several months, so it’s not 
a precise marry-up in that way. But I think JSF has very much 
reflected what we are seeking to do in the 5000. 

On the broader question, what DOD or the Congress needs to do, 
I think there are — the list is quite long in terms of the challenges 
we face. Part of it is the sort of entrenched cultures that we both 
have in terms of how we view programs and systems development, 
and some of the issues that you raised with Mr. Rodrigues earlier 
today. And that’s both an internal, DOD problems with the Con- 
gress, and the external world. Part of it has to do with — and I’ll 
give you one example. 

I think one of the real challenges with the new 5000 rewrite — 
and this is not relative to JSF, this is a broader question — is if you 
are in this front-end process where you want to, and have to, dem- 
onstrate technologies, you want to also — and thus not make your 
commitment to EMD and, really, program commitment — until 
you’ve gotten to that point, you will also hopefully, as we go down 
this path, be looking at more options than you might otherwise 
have been looking at. And what that leads you to is having to 
avoid, or wanting to avoid, a program commitment either in the 
building or from the political environment before you’re ready to 
say, I’ve got a set of technologies that can get me this capability, 
and that’s when I’m ready to make that commitment. 

You know, traditionally we’d say, we’re going to do a new jet or 
we’re going to do a new ship, and then we sort of work it through. 
But the ship never left the table or the plane. It was fairly rare 
once you got that far down the path. 

What we really want to do is both for ourselves inject a great 
deal more discipline. As I said, we have tremendous agreement, 
and GAO has been a partner at the table as we have built this 
model, but we also have to have discipline in the process to realize 
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that we may want to be looking at different kinds of capabilities 
and different kinds of mission options before you make those kinds 
of commitments. And that’s traditionally been very difficult, I 
think, for all of us. 

So there are a number of challenges that we all face in moving 
this forward. And I think that they’re very — I think they’re chal- 
lenges that we can overcome. As I said, I do believe that JSF is 
really the forerunner coming out and applying many of these prin- 
ciples. 

The General was talking about the process they went through 
with the various stakeholders, the different services. A really flexi- 
ble requirements process driven by what the customer really need- 
ed and what we believed as we went through the process tech- 
nology was going to be able to provide in a reasonable period of 
time, so that we didn’t end up waiting 18 years or 20 years, as has 
been the case in the past. 

Mr. Halloran. Finally, let me ask a followup where Mr. Rapallo 
began. 

Would you agree that all of the impacts you cited that might flow 
from a delay at the demonstration evaluation phase can potentially 
be more serious, more costly and lengthier if they’re incurred in the 
next or subsequent phases of the program, that, as is described — 
we may disagree on the $1 to $100, but there is an almost inevi- 
table escalating effect in delaying problems in this process. 

Mr. Soloway. That would certainly be the conceivable output — 
outcome. However, I think our view at this point is that the 
delay — to arbitrarily decide today to delay, to really do, in es- 
sence — and I don’t want to get into nitpicking, but in essence what 
we’re talking about here when you get to level 7 is the early phases 
of EMD. 

That’s what we do in engineering. We focus on the manufactur- 
ing part and the production part of it. We don’t focus on the “E”, 
which is the engineering. That’s a critical next step in bringing 
these technologies together. We believe, and we could be proven 
wrong next year and the decision would not thus be made, but we 
believe the technologies will be at that proven level that enable you 
now to move into that engineering of EMD phase. 

So given that, the delay to us would be costly, unnecessarily cost- 
ly, and would be just unnecessary on the face of it. But in a hypo- 
thetical world having nothing to do with JSF, certainly if you have 
tremendous technology issues and you go into EMD and lock your- 
self in, you could be asking for an escalated problem. I wouldn’t 
disagree with that; I just don’t think that applies to the JSF. 

Mr. Halloran. Thank you. 

Mr. Shays. Let me say, then, I would like David to come back 
with answering a question related to what you were saying up 
here. 

I’ve heard you both say, you know, this is something, you know, 
we have never done before. You’re very proud of the program, the 
fact that you’re moving toward a best commercial practices. But 
what I feel like, as I hear you, is that it’s going to be so significant 
to DOD that you’ve moved in this area if you do it halfway, a 
three-quarters way, this is monumental. And I might agree with 
that, that it would be monumental, but I have to hear an argument 
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that says you can’t go all the way with the program. And so for you 
to say you’re doing three-quarters of it — and you haven’t used those 
numbers, but that’s the feeling I get — wouldn’t satisfy me unless 
you say we can’t do best commercial practices here, here and here 
because, and then let us evaluate it. 

So I am going to stay on the table that really there is no argu- 
ment against using best commercial practices to the nth degree, 
unless, and I just kind of want to say that to you. 

Mr. Soloway. If we gave that impression, I think both of us 
would regret it. How do you define doing it all the way versus the 
three-quarters? 

Mr. Shays. Well, when you start to talk about it as a tool and 
not a requirement. And maybe in the private sector it’s a tool and 
not a requirement, and then that’s fine. 

Yes? 

General Huot. I would just say that — I would emphasize that we 
use a different tool. And again the reason we did it is because we 
wanted something that would allow us to manage risk, not just as- 
sess it. And our intent is not to go three-quarters of the way; we 
intend to reduce every critical technology that we identified to a 
low level of risk before we go into EMD — very important point. 

Mr. Soloway. And the requirement that is there is the tech- 
nology maturation requirement. That is a requirement; that is not 
a negotiable. 

What is negotiable is the TRL becomes like the yardstick, so it’s 
versus using a yardstick or a ruler or another method of measuring 
what the outcome is going to be. But the key here, the outcome we 
all seek is technology maturation. And that is not a negotiable. 

Mr. Shays. Does the DOD 5000 rewrite follow this practice, best 
commercial practices? 

Mr. Soloway. Yes. 

Mr. Shays. Because when you say it, you say in your statement 
on page 10, “Finally, let me be very clear. The strategy I have ar- 
ticulated for the revised acquisition process that will be prescribed 
in the DOD 5000 rewrite, and which is largely being implemented 
on the JSF, represents a real departure from our traditional ap- 
proach to system development.” 

That’s the basis for my making that comment. I buy into the fact 
that it represents a real departure from our traditional approach 
but it’s still the word “largely,” you know. So just so you have a 
sense of why I get the feeling I get. 

Mr. Rapallo, you had a question. 

Mr. Rapallo. I just had one followup on Mr. Halloran’s questions 
about the TRL levels. 

You said — I’m just trying to understand. The information that 
was provided to GAO was from the contractors, their evaluation, or 
from the program office? 

General Huot. You know, I wasn’t involved in that process, but 
I know the program office is involved with the contractors. 

Mr. Shays. I didn’t hear the question. I would like to ask you to 
ask it again. 

Mr. Rapallo. It was just a question of where the information 
came from, the TRL levels, from the contractor or the joint pro- 
gram office, or probably some combination of both. 
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Mr. Shays. So what is the answer? 

[The information referred to follows:] 

The input for the TRLs reflects a consolidated answer from the JSF Program Of- 
fice, incorporating input from the Boeing Company and Lockheed Martin Corpora- 
tion. 

General Huot. I believe it is, Mr. Chairman — I wasn’t involved 
in the exact process. I know the program office was involved, but 
I think the contractors actually did the work either with the pro- 
gram office or the oversight of the program office. 

Mr. Rapallo. And your position was that the GAO asked for in- 
formation based on TRL levels, asking also, including the risk of 
integration? 

General Huot. Yes. 

Mr. Rapallo. That information was provided? 

General Huot. Yes. 

Mr. Rapallo. And that would be incorrect, you’re saying? Basi- 
cally, I’m trying to figure out if these numbers are too low because 
of that. 

Mr. Soloway. That’s the crux issue here, that if you measure in- 
dividual technologies relative to the risk of being able to integrate, 
or to integration, you come up with one answer. If you measure 
them as individual technologies relative to their technology matu- 
rity, you come up with the second. And that’s what the rescoring 
was. 

So in our view, the rescoring actually does raise the numbers, not 
to level 7, but they certainly raise it. 

Mr. Rapallo. Are these the numbers that represent the rescor- 
ing or the initial? 

Mr. Soloway. These numbers are the initial. 

Mr. Rapallo. So the rescore numbers would be higher than this? 
The question is what are they I guess. 

Mr. Soloway. I think all of them would be at five and six. 

Mr. Shays. I would like you to tell us what would they be at. 

Mr. Soloway. I don’t know if I have the specifics on each of 
them. 

Why don’t we take that for the record and give you — I’m sorry. 
OK. These are our estimated ratings. 

Mr. Shays. Based on not having integrated? 

Mr. Soloway. Right. These are what was referred to as “the re- 
scored numbers.” 

Mr. Shays. Without the risk of integration? 

Mr. Soloway. Right, they were not scored. We have — and I’ll 
just take them in order — 7, 6, 6, 7, 6, 7, 6, 7. 

Mr. Shays. OK. 

Mr. Soloway. And I have the rationale, if you just — I mean, I 
can tell you what the technology — but in order of these numbers, 
here again, as — I think General Huot referred to this a moment 
ago. One, flying on the concept demonstrator aircraft, which is we 
believe consistent, technologies already flown in aircraft helicopters 
or spacecraft, another one, such systems are similar or being used 
in similar commercial environments, including the Boeing 777, 
Federal Express Caterpillar, etc.; another one flying on the F-16; 
another technology flown on the JSF flying test bed; another one, 
the same, flying test bed; another one, flown on the JSF flying test 
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bed; and another one, flying on the CDA as well as the F-22, the 
F/A-18 and the Eurofighter. So in each of these cases, if you look 
at are we using the technology in a relevant environment, per se, 
they are all being demonstrated where — in that means. 

Now, that is the rescored numbers. 

Mr. Shays. But that’s based on your assessment, correct? 

Mr. Soloway. It’s a DOD assessment. 

Mr. Shays. Right. And so in technology 7 you have a leap of two 
levels just by not having to integrate. And technology 1 you leaped 
up one, in technology 2 you went up two, technology 3 you went 
up two. In technology 4 you went up two. In technology 6 you went 
up two. In technology 7 you went up two. That’s a big jump. 

General Huot. Again, because you’re trying to mature the tech- 
nology without considering the risk of the integration on the JSF 
platform. 

Mr. Shays. So we can basically accept GAO’s assessment that 
they got — if we have a footnote that says it’s not integration, based 
on — excuse me, based on that, and we could accept your numbers 
based — excuse me, the other way around. 

They’re saying it’s not integrated; therefore, that’s the score. And 
you’re saying, if you don’t consider — I’m going to say it right here. 

Mr. Soloway. I think I understand where you’re going. 

Mr. Shays. For my own sense of self-esteem, I need to say this 
again. 

The bottom line is, GAO gave you a lower score because of a fail- 
ure to integrate. And you’re saying integration shouldn’t have been 
a factor in your score, and your score is higher without integration. 

Mr. Soloway. You’re asking if we can accept this as — under that 
other definition, if you will, and I think I would have to say I don’t 
believe we can because there are other issues associated with it. 

Mr. Shays. You want a little conference here? 

Mr. Soloway. We might want to take it for the record. But the 
table itself is confusing, and I’m not sure that we would precisely 
accept every level that was there. 

Mr. Shays. I welcome sometimes a little bit of qualification by 
two people in the same table. 

If that’s all right, General, do you want to just say what your 
sense is? 

General Huot. I think once again that you go back to where 
these things came from. They were provided to the GAO, but those 
scores were based on considering the risk of integration on the JSF. 
When those scores were rescored again and provided to the GAO, 
they were without integration of JSF, the integration on the JSF, 
that’s when they go up to the appropriate level. 

Mr. Shays. But I just want to know if you accept their score 
when there’s — by — and look at the score that you have provided us, 
would they accept your score, in your judgment? I mean, in your 
talks with each other, was this really the big debate whether it was 
integration or not and you could pretty much agree on your num- 
bers? 

OK. The answer is hard to figure out what they were saying 
when they did this, right? 

Mr. Soloway. Let us take that for the record, sir. 

[The information referred to follows:] 
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The information follows: 

The TRL levels of the highest risk areas, both the initial levels 
and the re-scored ones, are consolidations of JSF Program 
Of f ice/Boeing/Lockheed Martin input. Originally the GAO's 
supplied ground rules were understood by the JSFPO to require 
scoring TRLs at levels including both technology maturation and 
the integration risk. Subsequently, based on the draft GAO 
report, the program office gained further insight into ground 
rules for scoring, and re-scored the technology areas using just 
technology maturation risk. When solely the technology 
maturation risks are addressed, the 8 highest risk technology 
areas are rated as either TRL 6 or 7 at the start of EMD, which 
is acceptable for entry into EMD. The Department agrees that it 
is desirable to mature technologies to TRL 7 prior to entering 
Engineering and Manufacturing Development. However, the 
Department also feels that while TRLs are an important input, 
they are not sufficient alone to decide when and where to insert 
new technologies into weapons system programs. TRLs are not a 
tool used by the JSFPO to manage risk; instead the program office 
uses risk waterfall methodologies, which yield a greater amount 
of information about technical maturation processes. 
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Mr. Shays. Fair enough. You all have been very responsive. And 
is there a question you would have liked me to ask that you pre- 
pared all night for that you want to — I’m just going to say my staff 
gave me so much to read I stayed up all night. The statements 
have been very interesting on the third panel, as well as your pres- 
entation has been very, very helpful and, I think, candid. 

Mr. Soloway. I’d just add one point, it’s not a question, and that 
is to go back to what we said earlier. 

The debate here is not over whether or not we do need to dem- 
onstrate and whether or not we’re committed to demonstrating 
technology maturity before we move into EMD. And that is a very 
important step in a critical discipline that we are instituting in our 
process with the JSF and in future programs. And I believe it’s 
very important, as we do that, to recognize what it takes and to 
recognize the “E” engineering part of EMD, which is the next step. 

So I would just reiterate that point again. 

Mr. Shays. I don’t know if I’ll be chairman next March but it will 
be very interesting. One of the things we do is, we put every state- 
ment on the record; then we have to live with it. I’m just — the 
problem is, when it comes to this kind of program, the same people 
aren’t always the same people that have to answer for the decisions 
that were made 2 years ago — I mean on the congressional side as 
well as your side. 

Mr. Soloway. Sir, I just point out I won’t be here next March; 
that’s a guarantee. I think you saw the news this morning where 
both the House and Senate Armed Services Committees have in- 
cluded language requiring us to be able to demonstrate to the Con- 
gress that we have achieved technology maturity before we move 
into EMD. And we can live with that and have no problem with 
coming back to demonstrate that or document that for the Con- 
gress next year. 

Mr. Shays. That means low-level risk. 

Mr. Soloway. Thank you. 

Mr. Shays. General, any comment you’d like to make? 

General Huot. No, I think that covered it. 

Mr. Shays. We’ll end on that note. Very nice to have you both 
here. Thank you. 

We’ll get to our third panel. I appreciate the patience of our third 
panel. I have three panelists, if you would remain standing so I can 
swear you in: Dr. Thomas McNaugher, deputy director, Arroyo 
Center, RAND; Mr. Rodney Larkins, business development man- 
ager of 3M Corp.; Dr. Wesley Harris, Department of Aeronautics 
and Astronautics, Massachusetts Institute of Technology. 

If the three of you would — we’ll make sure we get you there. If 
you would raise you right hands, please. Thank you. 

[Witnesses sworn.] 

Mr. Shays. Note for the record that all three have responded in 
the affirmative, and we’ll just go down the row. 

Doctor, we’ll start with you first. And actually we have two doc- 
tors here. I’m sorry. 

Dr. McNaugher. 
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STATEMENTS OF DR. THOMAS L. McNAUGHER, DEPUTY DIREC- 
TOR, ARROYO CENTER, RAND; RODNEY LARKINS, BUSINESS 
DEVELOPMENT MANAGER, 3M CORP.; AND DR. WESLEY HAR- 
RIS, PROFESSOR, DEPARTMENT OF AERONAUTICS AND AS- 
TRONAUTICS 

Mr. McNAUGHER. Thank you, Mr. Chairman. It is a pleasure in- 
deed and an honor to be here. At the risk 

Mr. Shays. I will ask you to speak a little louder or move the 
mic a little closer. Can you pull it a little closer? 

Mr. McNaugher. Somewhere between last night and this morn- 
ing I have acquired a cold, so I am about an octave lower. 

Mr. Shays. I have absolutely no sympathy for you. 

Mr. McNaugher. Let me start by saying I don’t know much 
about the JSF program per se. I was asked to speak here because 
I think I know a great deal about the weapons acquisition process, 
at least as it functioned during the cold war. I wrote a book about 
that. It came out in 1989 from the Brookings Institution. And in 
that book, called “New Weapons, Old Politics,” I examined what 
might be called some of the perverse patterns of cold war weapons 
acquisition, one of which was the tendency to rush weapons 
through development and into production. 

In particular, I focused less on the move from what’s called the 
“demonstration-validation phase” and more on the move out of 
what is now called EMD and into production, so-called “production 
concurrency.” So it’s not exactly the same issue that you’re dealing 
with today although I think you’ll find a lot of what I have to say 

about the incentive structure and the way we measure 

Mr. Shays. Could you just suspend for 1 second. I’m sorry. 
Thank you. 

Mr. McNaugher [continuing]. Production concurrency or the 
tendency to substantially overlap the latter stages of development 
with the early stages of a move to high-rate production was a fairly 
common practice during the cold war. It was justified on two 
grounds. Technically it was justified on what might be called the 
“theory of declining uncertainty.” back in the “dem-val” phase. In 
a somewhat linear fashion, we reduce those uncertainties during 
development until, by the end or nearing the end of EMD, we 
should be able to move into production. 

On the other hand, on strategic grounds we were confronting a 
numerically superior enemy, so we justified the rush to production 
as a way of getting the jump on the Soviet Union, getting ahead. 
The faster you could get these new technologies out, the farther 
ahead of the Soviet Union you were, everything being equal. 

Now, when I started my book, I more or less agreed with those 
premises. When I finally started writing, I had come to disagree, 
especially with the first one, the theory of declining uncertainty. I 
would substitute what I call the “J-curve theory of program uncer- 
tainty.” True, at the beginning of a development project you have 
enormous risks, and you do, through “dem-val” and in the early 
part of EMD, neck those down to reasonable levels. But I encoun- 
tered in almost all programs a rather sharp upturn in risks as the 
program actually went into production. 

Three reasons for that: One was systems integration. And I like 
the fact that you’re talking about systems integration this early in 
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the program. During the cold war systems integration often oc- 
curred very late in the development program, sometimes after pro- 
duction systems were already out in the field. And what you find, 
looking back at the history, is that even simple components stuck 
together in a new way produced new problems that you didn’t 
know were there, these problems often force a design iteration. 

The second problem that drove risks up was the move to produc- 
tion tooling itself. Again, even with simple systems, somehow the 
move to a production model from a preproduction prototype intro- 
duced risks, some of them great, some of them small, which almost 
always led to a subtle interaction between the producing engineers 
and the developing engineers. 

And finally, when you got this thing out into the field where real 
soldiers and sailors and pilots could use it, they invariably discov- 
ered new ways to use it that it wasn’t designed to do, and it started 
to break, and then you would have to go back and redesign it and 
fix those. This is, in a sense, a tribute to the ingenuity of devel- 
opers and also users in the military, but it could be pretty embar- 
rassing sometimes, the kind of breakage you could get. So all three 
of those then would appear very late in the process. 

To the extent that production concurrency ignored those risks, 
and to use the jargon of today’s hearings, to the extent that, in a 
sense, it wasn’t “knowledge-based,” it didn’t take account of these 
late-arising risks, we had chosen a fairly expensive way of develop- 
ing weapons during the cold war. If you’d geared up for production, 
capitalized, and hired the labor, every delay, every change, cost a 
lot of money. If you already had systems in the field you were faced 
with this vexing question, do we retrofit the fixes to the fielded sys- 
tems, which was always expensive, or do we just let those go? This 
led to what I would call the ABC approach to development. 

The A model of a lot of airplanes, and other things, too, often had 
serious design anomalies in it; the B model got it roughly right; 
and the C model is where you really wanted to be all along, but 
it took maybe even 100 or 200 production models to get there. So 
it was expensive. And when you did that, the average effectiveness 
of our overall force actually was lower than it would have been had 
you waited and gotten all of the production run, or most of it, to 
the C level. 

Now, we accepted those costs ostensibly in a desire to confront 
the Soviet Union. And you would think that now with the cold war 
over, as you say, Congressman Shays, we could relax and we could 
do this better. Let me make clear that the way you do it better is 
not by having a sharp divide between development and production. 

The point of my research was that the early stages of production 
are part of the development process. Again, to turn to the jargon 
of today, knowledge points 2 and 3 tend to fudge together and you 
don’t want to have a sharp gap. Rather, you had to treat the early 
stages of production in a way that with respect to late-arising un- 
certainties, perhaps starting very low, with a low rate of produc- 
tion, getting the systems out into the field, flying or shooting or 
driving the dickens out of them for awhile, and taking all of that 
information back and imparting it to your design before you really 
went to high-rate production. 
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One of my colleagues at RAND years ago remodeled some of the 
Air Force aircraft programs of the 1960’s, 1970’s and 1980’s on that 
basis and concluded that you could probably get a more effective 
fleet overall for a little less money if you did it that way. Now, you 
added a little bit to the development cycle, which everybody 
thought was too long, but with the cold war over, that shouldn’t be 
much of a concern. 

So we might be in a position to move now to a more relaxed ap- 
proach. I’m skeptical of our ability to do this, however, because I 
think that production concurrency was as much a political as a 
military strategy; that is, it was rooted in the politics that we see 
here today — in the politics of the acquisition process. 

Surely in the development of a new system, the move into late 
development and early production has to be the most vulnerable 
stage. You now have prototypes of the system, you fly them, drive 
them; the data almost always is going to contradict the optimism 
of early assessments. Sometimes you get tragic accidents, a heli- 
copter crashes or a plane crashes. Sometimes you get funny acci- 
dents, you know — the gun that’s supposed to shoot a helicopter 
shoots the fan on a nearby latrine, as was the case with the Armys 
DIVAD. 

If this were purely a technical environment, everybody would 
have a good laugh, and they’d go back and redesign that system 
and fix it. But in the very charged political environment that can 
come to surround expensive programs, you know, it’s hard to de- 
sensitize that evidence. Knowledge and information become very 
dangerous. And delay — I’m thinking of your earlier discussion — you 
cannot convince a program manager that a 6-month delay isn’t the 
beginning of a mortal wound. 

How do you handle that? Well, in a sense you stack the deck. 
You raise the costs of slowing down. And you may even have pro- 
duction versions out there before you actually get that information. 

What I’m describing, in a sense, is the dilemma of weapon acqui- 
sition in this country as a political as well as a technical process. 
From a technical point of view, you really want to have a great 
deal of flexibility late in the process. From a political point of view, 
flexibility can be downright dangerous. So we tend to structure a 
certain amount of inflexibility in there. 

This is not a dilemma we have ever resolved very well in our his- 
tory. If you step back from the cold war and look at the 200 years 
in which this Nation has bought weapons, you would argue that 
the problem wasn’t slowing the acquisition process down, it was 
getting it to produce anything at all. We would test endlessly and 
then not buy. Our technology generally lagged the Europeans, and 
as a rule, we had trouble getting things through the juggernaut of 
the political process. That ended with World War II and with the 
cold war. 

So while talking about political strategies may sound like I’m 
being critical, we ought to understand that for 50 years we pro- 
duced and systematically modernized the best force posture in the 
world.? So it’s not clear to me that we can slow this thing down. 
And this is why I refuse to pass judgment on the patterns of acqui- 
sition I studied in the cold war. 
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We are now in a unique period in the Nation’s history. We are 
not confronting a great Soviet threat, as you’ve said, although the 
world remains dangerous; moreover, we remain engaged, and there 
are threats out there. So we’re sort of at the Goldilocks situation, 
you know, it’s not no threat, not a big threat, but sort of right in 
between. And the question is, can we come to the Goldilocks solu- 
tion for weapons acquisition? That is a process that is relaxed 
enough to take account of late-arising uncertainties, but not a proc- 
ess that goes to sleep and doesn’t produce anything. 

And that returns me really to the first paragraph of your state- 
ment, Mr. Chairman, which talks about the emergence of a consen- 
sus about how we’re going to handle this. 

This hearing focuses on the JSF specifically, but the JSF, as you 
recognize, is the beginning of a wave of new modernizations and re- 
capitalizations after a 10-year procurement holiday. So this hearing 
is also part of the way in which we construct this consensus, and 
it remains to be seen how that will work out. 

Thank you. 

Mr. Shays. Thank you very much. Your book was the reason why 
you’re here and your testimony justifies your presence. 

[The prepared statement of Mr. McNaugher follows:] 
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A decade ago I published a book about military research, development, and acquisition’ 
in which I identified patterns of defense acquisition that, while often producing some of the 
world’s best weapon systems, nonetheless represented a very costly way to develop and field 
new technologies, I am honored to appear before this subcommittee to discuss one of those 
patterns, namely so-called “production concurrency” - the widespread practice of rushing new 
systems through the later stages of development, into production, and thence to operating units. 
Let me be clear before I begin, however, that my message here is grounded in my book; I have 
done very little research on the weapons acquisition process since 1990. I am speaking for 
myself, in other words, not the RAND Corporation. 

Production concurrency was justified during the Cold War on three seemingly 
unassailable grounds: 

• First, moving fast seemed technologically feasible because, so the theory went, most of the 
uncertainties associated with developing new technologies were resolved early in the 
development process. By the time the developer neared production, there were few 
technological hurdles left to clear, hence no reason not to move fast. 

• Second, moving fast seemed strategically desirable, even necessary, for a country that had 
chosen to rely on technological advantage in confronting the Soviet Union’s numerically 
superior conventional forces. Almost all observers of the process agreed that the United 
States took too long - often 12 to 15 years -- to develop and field new weapons. Clearly, the 
shorter the development cycle, the farther ahead of the Soviets we would be. 

• Third, delays late in the development process were expensive, given that by then the 
developing firm had normally purchased or manufactured production tooling and hired a 
production labor force. 

Clearly the first of these justifications - what might be called the “assumption of 
decreasing uncertainty” - was key to the approach; the latter two justifications made sense only 
so long as technical challenges slowly disappeared in the latter stages of the development 


New Weapons, Old Politics; America’s Military Procurement Muddle (Brookings Institution, 1989). 
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process. Yet research for the book led me to question and ultimately reject precisely that 
assumption. What 1 documented was a significant upswing in technical and design uncertainty 
as programs neared the end of development. These were largely unavoidable, and sprang from 
several sources. 

First, “system integration” normally occurred fairly late in the development process, and 
presented unique developmental challenges and risks. Successfully testing individual 
components of a new system was no guarantee that those components would work together as a 
system. The component parts of the B-1B bomber’s electronic countermeasures (ECM) suite, 
for example, tested well individually. But when placed in position aboard the aircraft they 
interacted with each other in perverse ways that demanded redesign of components as well as the 
system as a whole. Significantly, in this case system integration occurred very late indeed; the 
ECM suite was not tested as a system until after the first production B-lBs entered operational 
service. 


System integration problems seemed to plague even relatively simple combinations of 
reasonably well-understood components. The Army’s DIVAD (division air defense) system 
mated the old M-48 tank chassis with the F-16’s APG-68 radar and a production model German 
gun system. Not a difficult combination, one might think, and on that assumption that Army’s 
program office sought a relatively rapid move to production. Indeed, the Army signed a fixed- 
price final development and production contract with Ford Aerospace, the winning contractor, 
that raised the costs of violated contractual test and production milestones. Yet the Army was 
forced to violate those milestones, as DIVAD encountered an array of technical and operational 
test difficulties.that ultimately led to its cancellation in 1985, Not all of DIVAD's problems 
stemmed from systems integration; getting an aircraft radar to work in the ground environment 
proved to be more challenging that expected. But integrating DIVAD’s three basic components 
contributed to the system’s problems and ultimate cancellation. 

The move to production tooling inserted a second set of new technical challenges into the 
acquisition process, and often ushered in a subtle interaction between the original design and the 
tooling for it. As with DIVAD, this observation applied even to seemingly simple systems. 

Army contractors successfully prototyped the TOW (tube-launched, optically-tracked, wire- 
guided) ami-tank missile in the early 1960s, but the system did not enter production until 1970. 
The delay stemmed in substantial part from technically difficulties associated with mass- 
producing spools of very thin wire wound carefully enough to unravel quickly, as the missile 
flew, without breaking. 

A third set of late-arising technical challenges emerged when new systems were sent to 
operating units. Sometimes rigorous, day-to-day operational use surfaced lingering design flaws 
that remained hidden during formal operational testing late in the development process. But the 
more pronounced design challenge normally stemmed from the extent to which real operators - 
pilots, soldiers, and sailors - use new equipment in ways that were not envisioned by designers 
or tested in formal testing. Although this was a tribute to the work of developers - their systems 
invited novel uses - it nonetheless could produce embarrassing and costly technical problems in 
their products. Some of the reliability and performance problems that plagued the US Air 
Force’s F-1000 engine, for example, stemmed from the engine’s remarkable power and 
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resilience. Whereas the engine had been designed principally for speed, pilots of the new F-15, 
the first aircraft powered by the F-100 engine, found operational advantage in rapidly changing 
speed, submitting the engine to “thermal cycles” not envisioned as the engine was designed and 
tested. This was one of several reasons the new engine underwent nearly a decade’s worth of 
maturational development before reaching its full potential. 

These three sets of technical challenges were, in my view, more-or-less genetically 
associated with the development of new weapon systems. A fourth set of challenges appeared to 
be avoidable. This set resulted from major system redesign at the end of the early phase of 
development (the so-called “demonstration-validation,” or “dem-val” phase) that substantially 
devalued the information gained during that phase. In these cases, what was then called “full- 
scale engineering development” (FSED, now called “engineering-manufacturing development,” 
or EMD) confronted risks and technical challenges more appropriate to an earlier phase of 
development. 

Redesign was sometimes prompted by the failure of technological prospects in the dem- 
val phase. In the development of the Air Force’s Advanced, Medium-Range Air-to-Air Missile 
(AMRAAM), for example, dem-val did exactly what it was supposed to do - it showed that new, 
solid-state components could not produce the radar power required of that missile. FSED thus 
commenced on the basis of a more traditional design based on standing-wave tube technology. 
But getting the required power from that older technology into a tube small and light enough to 
be carried by the F-16 tossed up enormous operational and technical problems that drove the 
system’s costs up and plagued early operational testing. Ultimately the Congress intervened 
directly in the program, setting out test, cost, and scheduling criteria of its own to control 
AMRAAM’s late development. 

On other occasions, the services imposed significant new and additional requirements on 
the evolving design simply because the move from dem-val to FSED presented the opportunity 
to do so. The F-16’s dem-val phase was the so-called “lightweight fighter competition,” held in 
the early 1970s, which pitted prototypes from General Dynamics and Northrop against each 
other in a competition focused exclusively on fighter characteristics. In taking the winning 
design (General Dynamics’) into FSED, the USAF asked for strike (bombing) as well as fighter 
capabilities. Not surprisingly, the additional requirements prompted substantial design changes 
in the F-16’s airframe as well as its avionics. 

Production concurrency represented a rather expensive way to structure the transition 
from development to production. Capitalizing and hiring labor on the basis of the optimistic 
assumption of decreasing technical risk raised the cost of every delay and design change. 

Moving into production before designs matured meant sending less capable systems into the 
force; a B-1B lacking a working ECM suite was not capable of performing the mission it was 
design to perform, namely penetrating Soviet airspace. These problems could be fixed, but it 
was more expensive to fix fielded systems than it was to fix the pre-production design. To the 
extent that the services simply foreswore expensive retrofits to fielded systems, on the other 
hand, they reduced the overall military effectiveness of their fielded fleet of that system. In these 
cases, production concurrency actually reduced the overall military effectiveness of the fielded 
fleet, while it also raised costs. 
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The pervasiveness of production concurrency in Cold War weapons acquisition suggests 
that the nation was willing to accept these costs, presumably in the urgent pursuit of continued 
technological superiority over Soviet forces. Arguably with the Soviet threat gone, the nation is 
in a position to slow the rush to production in an effort to handle the late-arising risks and 
uncertainties I have just described. Significantly, this does not mean imposing a sharp break 
between development and production. Rather, it means organizing acquisition in a way that 
recognizes that the move to production is in fact part of the development process, and that early 
operational experience with a new system is in a very real sense part of the system’s operational 
testing. One alternative - 1 suspect there arc others - would be a slower move to production, 
extensive operational use of initial production articles, and rapid feedback that takes the 
information so-generated and applies it to design “fixes” before the new system is far into 
production. 

Such an approach could yield considerable military effectiveness, possibly at reduced 
cost than what was associated with Cold War programs. Examining the F-15’s development 
schedule, for example, RAND's Allen D. Lee modeled an alternative approach that “minimized 
concurrency, extended low-rate production, and intensified testing [of initial production aircraft] 
and use of test information [to modify the initial design].” The result: 

Costs decreased about $7 million [in comparison to actual program costs incurred]; Air 
Force retrofit labor decreased by 180,000 man-hours; and overall [fleet] effectiveness 
increased modestly [because more F-15s were outfitted with larger fuel capacity and 
newer electronics technology]. 2 

Although slowing the rush to production to make these fixes would have added marginally to the 
length of the development cycle, this would be less of a concern in the absence of the Soviet 
threat. 

Notwithstanding these potential benefits, slowing the development process, and 
especially the move to production, may be very difficult to achieve. Production concurrency 
may have been rationalized on the basis of meeting the Soviet threat. But it served political as 
well as strategic purposes: it raised sunk costs and increased program “momentum” at a crucial 
juncture in the program’s evolution. In the sometimes highly charged political milieu through 
which weapon systems moved from inception to production, the latter stage of development was 
an especially dangerous phase. Elaborate testing inevitably yielded hard data that undermined 
the confidence of early - and almost always optimistic - cost and performance estimates. This 
was exactly what testing was intended to do, of course, and in a purely technical environment 
test data might have been used to “fix” the design. But in the political stew that often came to 
surround controversial or expensive weapons programs, bad test data could easily become the 
basis for political indictment, legitimate or not, killing or dramatically altering the program 
before it got to production. Production concurrency blunted these efforts by raising the costs of 
slowing down. Often production articles were already in the force before unflattering test data 
were available. 


2 Allen D. Lee, A Strategy to Improve the Early Production. Phase in Air Force Acquisition Programs, RAND P- 
6941-RGS, 1983. 
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I say this not to disparage the motives of service program managers but to call attention 
to the obvious, if under-appreciated, fact that America’s weapons acquisition process is a 
political, as well as a military-technical, undertaking. Defense funds are public money, and 
“national security” is a public good. Inevitably, incredibly complex technical development 
projects are submitted to the rigors of the annual budget process as well as countless other 
committee hearings. Bureaucratic machinations in the Pentagon interact with politics on Capitol 
Hill. Weapons projects become the objects of the “games people play” - the normal tugging and 
hauling by which U.S. public policy is made. 

Under these circumstances, one could at least argue that political strategies might have 
been essential to move new systems steadily through the political process. For much of its 
history the United States purchased weapons episodically if at all. In the realm of ground and air 
force equipment, the problem was not buying too fast, but buying anything at all! Normally U.S. 
weapons technology lagged rather than led that found in Europe. Against this background, the 
fifty years from the beginning of World War II through the end of the Cold War stand out as the 
only years in which the nation armed itself, more-or-less systematically, with the most 
technically sophisticated weapons in the world. Arguably it took a pervasive sense of threat to 
mobilize and focus the nation’s complex political processes sufficiently to achieve that sustained 
result. That result may also have required that programs be structured in ways that withstood the 
vicissitudes of the political process, even if doing so ignored some technological risks and raised 
costs. 


I make no value judgment either way about the politics of weapons acquisition or the 
behavior of program managers or other players in the process. 1 merely point out that, to the 
extent that the acquisition process is shaped by political as well as technical and military 
considerations, the collapse of the Soviet Union need not automatically usher in a new era of 
slower, more careful, weapon development programs. We can and should rewrite otir acquisition 
regulations to reflect an approach to development that respects late-arising risks. Hopefully an 
era marked by the absence of the Soviet threat yet continued U.S. engagement around the world 
will produce a political environment in which such regulations can affect programs more 
powerfully than they did during the Cold War. 

But this may be asking more than the political system can deliver, in which case the 
acquisition process will continue to be shaped largely by political forces. Concerned officials or 
legislators will focus on specific projects - the JSF being a good example — perhaps because 
they look risky, are expensive, exhibit excessive cost growth, or have failed certain tests. This 
was, I would argue, the Cold War reality. It can be arbitrary, especially in the short run. Over 
time, however, the political process does tend to pick out the losers, or the problem cases, and 
bring them under focused political control. Intervention of this sort still needs guidelines - a set 
of rules, or a model pattern of development, that can serve as the basis for imposed control. 

Thus here too it is worthwhile trying to write acquisition regulations that seek to prescribe such 
practices. But the extent to which these prescriptions are implemented will vary, perhaps widely, 
across programs. 

Which brings me to the subject of this hearing. Just as the Joint Strike Fighter is seen as 
the first major post-Cold War weapon development project, so this hearing should be seen as 
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helping to launch the post-Cold War acquisition debate after a decade long “procurement 
holiday”. It and others like it will play a role in shaping the political consensus t refer to above. 
Nowhere on the horizon can we find a numerically superior threat seeking to match or surpass 
our technological prowess across nearly the full spectrum of weaponry. Yet as a nation we 
remain engaged globally, and we have found our technically sophisticated weaponry to be very 
useful in handling problems from Iraq’s invasion of Kuwait to ethnic cleansing in Kosovo. This 
hearing and others like will have some influence on whether we can arrive at a political 
consensus that can support a weapons acquisition process that is more “relaxed” than Cold War 
practice yet not as lethargic and episodic as was the practice before World War II. 
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Mr. Shays. Mr. Larkins. 

Mr. Larkins. Good morning, or I guess I should say good after- 
noon. 

Mr. Shays. Good afternoon. You’ve been wonderful to be so pa- 
tient. But I learn as much from the third panel, and sometimes 
more. So from my standpoint, I’m very happy you are here. 

Mr. Larkins. Thank you. As business development manager for 
government research programs at 3M, I’m here to discuss 3M’s 
“best practices” associated with our new product commercialization 
process and the timely integration of breakthrough technology into 
“change the basis of competition” products. 

3M is a diversified manufacturing company with sales of more 
than $15.5 billion. We manufacture a broad range of products di- 
rected at six distinct commercial markets. 3M has grown by pio- 
neering innovative technologies and creating new products with 
these technologies, thereby creating new markets and revolutioniz- 
ing existing ones. 

3M has a written policy objective which states that 30 percent 
of all product sales for a given year come from products that were 
introduced in the 4 preceding years. Thirty-four percent of 3M’s 
total sales in 1999 came from products in this category. It is these 
new products which help sustain the profitable growth of our cor- 
poration. To maintain this rapid pace of technology development 
and new product introduction, 3M has evolved a well-defined tech- 
nology development and commercialization process. 

A basic tenet to successfully introducing new products is the dis- 
cipline to make the up-front investment in research and develop- 
ment. 3M has invested more than $1 billion per year over the past 
3 years in research and development. This investment is directed 
both at broadening and strengthening 3M’s existing technology 
portfolio, as well as moving products rapidly through the commer- 
cialization process. 

Currently, approximately 20 percent of 3M’s research and devel- 
opment budget is directed at technology development and enhance- 
ment. Approximately 80 percent is directed at product scale-up and 
commercialization. This investment ensures the availability of criti- 
cal technologies to turn into products, and the product development 
resources to focus on the corporation’s priority programs. 

In our quest to maintain technical and market leadership in the 
markets we serve, we have evolved into a laboratory structure 
which substantially segregates technology development from prod- 
uct development. Our experience has been, that the key tech- 
nologies are used broadly across the corporation’s market groups. 
By focusing on technology development in a technology center, a 
critical mass of technology experts is created, and costly redun- 
dancy is eliminated. 

3M has formally established 14 technology centers for the cor- 
poration’s most pervasive technologies. The responsibility of these 
centers is to establish and maintain world-class capabilities in 
these critical technologies. The second responsibility is to work 
with product development teams in the business units to integrate 
this technology into the product development programs to assure 
successful and timely product introduction. This step is critical, 
and a number of management tools have been established to en- 
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sure that it takes place. 3M is acutely aware that even our best 
technologies, when not applied to timely commercial product devel- 
opment, are of no value, and we work hard to maximize that appli- 
cation. 

Critical to the commercial success of our company is our ability 
to select and to focus on programs with the best possibility for 
changing the basis of competition in large market opportunities. 
Despite 3M’s large investment in research and development, our 
resources are finite and there are always more opportunities than 
there are resources. Important steps in program initiation are op- 
portunity identification, development of a comprehensive product or 
system description, a business assessment of projected results once 
success has been realized, and a technology assessment which indi- 
cates that technologies are in place to meet performance require- 
ments. All of these elements must be in place before a program is 
approved for scale-up and commercialization. 

At the initiation of the product development process, a formal re- 
view is held to assure that all elements are in place for success. 
Hence, success on these key programs is critical to the success of 
the corporation and our goal is to put these programs in the very 
best possible position to succeed. 

Once under way, the development team follows a detailed new 
product introduction plan which outlines all elements which must 
be accomplished during the product development process. Periodic 
program reviews are routinely conducted with the emphasis placed 
on encouraging and rewarding candor on the part of the product 
development teams and identifying and eliminating impediments to 
the success of the team. 

In addition, corporate review teams are employed. These review 
teams bring together experts in product development from across 
the corporation to identify potential roadblocks to successful com- 
mercialization and to identify resources to eliminate those road- 
blocks. 

In summary, 3M’s approach to technology integration and prod- 
uct development has proven to be very successful. This process, 
however, is far from being perfect. It’s a process that we will con- 
tinue to evolve, and our goal, of course, is 100 percent success on 
all of our priority programs in the corporation. 

Thank you, Mr. Chairman. I’ll be happy to answer any questions. 

[The prepared statement of Mr. Larkins follows:] 
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Good morning Mr. Chairman and members of the Subcommittee. Thank you for 
the opportunity to testify before you today. 

As the Business Development Manager for Government research programs at 3M, 
I am here to discuss 3M “best practices” associated with our new product 
commercialization process and the timely integration of breakthrough technology 
into “change the basis of competition” products. 

As you may know, 3M is a diversified manufacturing company with sales of more 
than $15.5 billion. We manufacture a broad range of products directed at six 
distinct commercial markets. 


1 . The Industrial Markets Group markets products such as masking tapes and 
abrasives. 

2. The Transportation, Graphics and Safety Market Group markets products 
such as reflective sheeting for traffic signs and respirators for worker safety. 

3. The Health Care Market Group markets products such as advanced drug 
delivery systems and health information systems software. 

4. The Consumer and Office Market Group markets products such as Post-it® 
Notes and Scotch-Brite™ scouring pads. 

5. The Electro and Communications Market Group markets products such as 
3M™ MicroFlex Circuits and 3M™ Volition ™Fiber Optic Cabling System. 

6. The Specialty Materials Market Group markets products such as Dyneon™ 
Fluoropolymers and Scotchgard™ Protective Chemicals. 
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3M has grown by pioneering innovative technologies and creating new products 
with these technologies; thereby, creating new markets and revolutionizing 
existing ones. 34 percent of 3M’s total sales in 1999 came from products which 
were new to the market within the past four years. It is these new products which 
help sustain the profitable growth of the corporation. 

To maintain this rapid pace of technology development and new product 
introduction, 3M has evolved a well defined technology development and 
commercialization process. 

Investment 

3M has invested more than a billion dollars per year over the past three years in 
research and development. This investment is directed both at broadening and 
strengthening 3M’s technology portfolio, as well as moving products rapidly 
through the commercialization process. Currently, approximately 20 percent of 
3 M’s R&D budget is directed at technology development and enhancement. 
Approximately 80 percent is directed at product development, scale- 
up/commercialization. In most cases, our company’s new products draw from 
3M’s base of established technology platforms such as adhesives, microstructured 
surfaces, etc. 3M has established a core of 34 technology platforms and are 
continuing to establish new ones on a strategic basis. Occasionally, 3M will 
license or purchase technology or use strategic partnering with another company to 
add a critical technology to enable a product to successfully meet the product 
description which is in place. 

Segregation of Technology and Product Development 

3M has evolved to a laboratory structure which substantially segregates technology 
development from product development. Our experience has been that key 
technologies are used broadly across the corporation’s market groups. By focusing 
technology development in a technology center, a critical mass of technology 
experts is created and costly redundancy is eliminated. 3M has formally 
established 14 technology centers for the corporation’s most pervasive 
technologies. The responsibility of these centers is to establish and maintain world 
class capabilities in these critical technologies. A second responsibility is to work 
with product development teams in the business units to integrate this technology 
into the product development programs to assure successful and timely product 
introduction. This step is critical and a number of management tools have been 
established to ensure that it takes place. 
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1 . Technology center executive management personnel are incorporated into 
market group executive management teams. 

2. Salary increases and promotions of technology center employees are 
substantially based on successful technology transfer leading to product 
introduction success. 

3. Transfer of technology center personnel, to product development/business 
group teams on either a temporary or permanent basis, is strongly 
encouraged and is, in fact, part of the technology centers’ charter. 

4. A fourth, critical element in assuring effective technology transfer into the 
business units (where product development activities occur) are the efforts 
expended in technology awareness programs throughout the corporation. 

For example, quarterly summary reports from all technology centers are sent 
to each pertinent business unit. Periodic technology center poster sessions 
are held where all 3M product development personnel are invited to interact 
with the individual scientists, ask questions about their work, and network 
on future product development programs. There is also a corporate 
technology annual event where the latest technology developments in the 
company are exhibited with the key technology experts available to answer 
questions. Our Technology Forum organization allows for this kind of 
interaction and communication to happen on a world-wide basis. 

3M is acutely aware that even our best technologies, when not applied to 
commercial product development, are of no value and we work hard to maximize 
that application. 

Initiation of New Programs 

3M develops and introduces many new products each year and a number of these 
are rather routine product upgrades and modifications which do not involve 
integration of new technologies. Our experience has shown, however, that by 
focusing on fewer, larger programs which have the ability to change the basis of 
competition in the market, a much more positive impact can be shown on corporate 
sales and profit results. I would like to focus my remarks on New Program 
Initiation on the process which has been developed for addressing these significant 
programs which are known at 3M as Pacing Plus Programs. While there is no one 
way a Pacing Plus Program is initiated, every case involves the identification of a 
customer problem or need and a product approach which can uniquely address that 
need. At this juncture, an assessment of magnitude of the business opportunity is 
made along with an assessment of technical and manufacturing feasibility. Based 
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upon the ability to create a successful business case for the program, a cross- 
functional team is organized including people from marketing, product 
development, manufacturing, and other disciplines. Their job is to create a detailed 
product description including essential and non-essential performance parameters 
(we call them musts and wants). Based upon this description, decisions are made 
as to whether (and which) new technologies are required to meet the product 
description. At that point, the team, in conjunction with the appropriate technology 
center, makes the determination as to whether the technologies can provide the 
necessary performance capabilities. In some cases, the answer is, not quite. In this 
situation, a risk assessment is done with a determination made as to whether the 
required technology development and integration can occur within prescribed 
schedule limits. It is important to exercise discipline, either to slow down or 
terminate a program where significant technology development work remains. 

This is critical because at this phase of the program it gets very expensive to 
proceed. It is best, at this point, to move the initiative back into the technology 
center for solution before proceeding. 

The Product Development Process 

Once a determination has been made (through a formal review process) that a 
business case can be made for the product and that early indications are positive 
that the product will meet performance and manufacturing targets, the product 
development process begins. During this period, a marketing and business plan are 
initiated, performance tests are developed, and a manufacturing process plan is put 
in place to bring the manufacturing process into control so that cost, quality and 
cycle-time targets can be met. Throughout this process, a series of management 
reviews are held to assess market status adherence to schedule, cost growth, etc. 
This phase of the new product introduction process incurs the most cost; 
consequently, the earlier critical problems can be identified and the program 
brought back on track, the better. One of the approaches for successfully 
accomplishing this is the use of corporate program review teams. These teams are 
staffed with experienced technology development, product development, 
marketing, manufacturing, etc. experts from across the corporation. The team’s 
goal is to help make the program be successful. There are, however, occasions 
where a program must be terminated or returned to the technology development 
phase because of inability to meet schedule performance or manufacturing 
parameters. 

Summary 

In summary, it is 3M’s belief and practices that in order to be successful, we must: 


+ 
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1. maintain significant investment in technology and product development; 

2. be preeminent in the technology platforms which support our major 
business; 

3. separate the technology development from the product development 
functions, 

4. effectively integrate advanced technology into our new product offerings; 

5. focus on fewer, higher impact programs; 

6. continuously refine the process for identifying and selecting key product 
opportunities for the corporation; and 

7. enhance the program monitoring process to provide the product development 
or commercialization team with required insight and resources or in specific 
cases, change the program direction. 

Again, thank you, and I’ll be happy to answer any questions. 


Rodney J. Larkins 
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Mr. Shays. Thank you, Mr. Larkins. You work for, I think, a 
pretty amazing company. 

When I think of your company I think of it as — though very 
large, it has tremendous innovation as a small company might 
have. And it’s great to have you here. I would love to know how 
we can seek comparables between what you do and what we need 
to do in government in general, and specifically with defense. 

Dr. Harris. I just want you to know, sir, I’m always awed when 
I see a “doctor,” and I see MIT next to it. 

Mr. Harris. Thank you. Good afternoon, Mr. Chairman. I very 
much appreciate the opportunity to address this distinguished sub- 
committee on 

Mr. Shays. I would ask you to move the mic a little closer to you. 
If you can turn it over the paper, as long as it doesn’t get in the 
way. Does it get in the way? OK. That’s perfect. 

Mr. Harris. I’m fine. Thank you. 

Again, I very much appreciate the opportunity to address this 
distinguished subcommittee on certain issues related to technology 
maturity and acquisition reform. I wish to state at the outset that 
I approach this important topic from the perspective of both an 
academic researcher and as a former government manager, not as 
an engineer practicing within the defense industry. 

It is my view that our defense acquisition policy and practices 
are complex. This complexity is a result of several factors, includ- 
ing a dynamic or shifting defense industrial base, a declining acqui- 
sition budget, a constantly evolving threat environment, a diverse 
force structure, and most importantly, an increasing rate of change 
of technology. 

The impact of the last factor, namely the increasing rate of 
change of technology, on technology maturity from an acquisition 
policy perspective, is difficult to overstate. The impact is made 
more profound when the global nature of much of this technology 
is considered in the acquisition of new weapons systems. 

I would like to add to this list the following elements that have 
impact on the acquisition of defensive systems: the current focus on 
greater life cycle value, the emphasis on more rapid deployment, 
the emphasis on upgradability, sustainment and maintenance. 

Mr. Chairman, while noting that today this subcommittee is ad- 
dressing a specific program within the defense acquisition arena, I 
wish to state for the record that the impact of acquisition reform 
reaches beyond the procurement of defense systems and its related 
technology. Through our defense industry base, acquisition reform 
drives our national economy and impacts world peace, in short, our 
success in developing an effective and efficient acquisition strategy 
that captures mature technology, exposes the risk to control our fu- 
ture, to produce wealth and to continue to contribute to the ad- 
vancement of humanity. 

Based on my research and government experiences, I wish to 
share with this subcommittee today several national successes. 
First, there exist several case studies of successful acquisition of 
defense systems in production. Second, there exist case studies of 
successful parallel development of advanced technology to high ma- 
turity levels where the government is customer. These two suc- 
cesses have many things in common, and I believe are related to 
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today’s issues of acquisition reform in the environment of tech- 
nology maturity. 

My success or my research on economically incentivized contracts 
focused on several important programs including the Sensor Fused 
Weapons system, the Joint Direct Attack Munitions program, the 
C-17 program, C130J, the F-414 engine, development that goes 
within the F-18E/F airplane and the F— 117 engine that goes in the 
C-17 airplane, as well as the Boeing 757 airplane. These programs 
were in production, were able to develop an economically 
incentivized contract or a win-win solution for both government 
and contractor. 

A few comments now on the development of a critical complex 
technology in advance of full system acquisition: In the early 
1990’s, NASA’s Office of Aeronautics did develop and manage two 
technology development programs. These were the high-speed re- 
search program and the advanced subsonic technology program. 
During the same time period, NASA also worked jointly with in- 
dustry and DOD to develop advanced gas turbine components with- 
in the integrated high-performance turbine engine technology pro- 
gram. 

These three programs of technology development were successful, 
and as stated in the prepared statement, over 12 reasons why they 
were successful. 

Mr. Chairman, the subcommittee may wish to note the very 
strong commonality between the factors leading to successful devel- 
opment of technology parallel to full-system acquisition and the fac- 
tors that enable a win-win solution for programs already in produc- 
tion. At the most fundamental level, the environment for favorable 
development of advanced technology is very similar to the environ- 
ment for acquisition of defense systems in production where tech- 
nology risk is low, corresponding to technology at a high maturity 
level. 

These programs, both advanced technology development pro- 
grams and acquisition of full defense systems, strongly suggest that 
advanced technology at a high maturity level is essential to the ac- 
quisition of affordable systems with requirements for superior per- 
formance. The importance of advanced technology at a high matu- 
rity level is so great, in my opinion, that the government must 
incentivize the contractor to develop advanced technology. 

This means that the government must place a premium on the 
development of technology to a high maturity level. The premium 
must compare favorably with other awards available to the contrac- 
tor. The economic realities of a high premium will, or at least 
should, drive the government to a lean portfolio management con- 
dition. Advanced technologies selected for development to high ma- 
turity level must be, or should be, based on realistic projections of 
need. 

Mr. Chairman before concluding, I would like to add a few addi- 
tional comments. I wish to note that to improve the chances of suc- 
cessfully developing advanced technology to a high level of matu- 
rity, one or we, those involved, should adopt common, quantitative- 
based language and assessment tools. Qualitative prescription of 
technology readiness, such as the technology readiness levels, are 
insufficient and inconsistent with realistic projections of need. The 
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desired quantitative-based language and assessment tools are rec- 
ommended to be derived from probability in each technology readi- 
ness level. The TRL would be expressed or defined by probability 
bands. 

In conclusion, we, as a Nation, have demonstrated the capability 
to produce weapons systems driven by advanced technology. My 
colleagues have clearly confirmed that. To become more efficient at 
the acquisition process, we must continue to develop advanced 
technology in parallel with acquisition of full, complete systems. 
Key elements again in this efficiency are to incentivize industry 
with favorable premiums, a realistic projection of advanced tech- 
nology needs, and the use of quantitative-based language and as- 
sessment tools. 

The bottom line is that government and industry know enough 
about each other and about advanced technology development to 
make affordable acquisition of full systems and technology inser- 
tion work, to make it work efficiently. The health and wealth of our 
Nation depends upon this working very efficiently. 

I would entertain questions, if there are any. 

[The prepared statement of Mr. Harris follows:] 
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TECHNOLOGY DEVELOPMENT STRATEGY 


Thank you for this opportunity to speak with you today. My name is Wesley L. Harris. 1 
direct the Lean Sustainment Initiative at the Massachusetts Institute of Technology where 
I am Professor of Aeronautics and Astronautics. The views I express are my own. My 
research on economic incentives for defense systems in full production has given me an 
acute awareness of the challenges of acquisition reform, many of which have been 
presented to and accepted by the Department of Defense (DoD) and the U.S. defense 
aircraft industry. My former duties and responsibilities as Associate Administrator for 
Aeronautics, NASA, included the management of the parallel development of 
technology critical to the advancement of new, high performance civil aircraft. I believe 
that this experience at NASA is directly relevant to providing an informed perspective on 
the central issues being addressed in this hearing. 

1.0 Introduction 


Acquisition reform for the procurement of military systems and technology is driven by 
several factors, many of which are interrelated in a complex manner. These include 
greater life cycle value, higher performance, more rapid deployment, upgradeability, 
sustainment and maintenance. The challenges confronting acquisition reform in the 
procurement of military systems are sharpened by the pronounced decline in the DoD 
procurement plus research and development outlays from a high of $150 billion in 1983 
to a low of $50 billion in 1998 (measured in FY99 dollars). 1 The changing characteristics 
of the threat to national security have added to the difficulty of developing and remaining 
committed to a consistent acquisition reform strategy. The very high rate of change of 
technology, especially information technology, remains a singular almost insurmountable 
challenge to the acquisition of advanced commercial and military systems with superior 
performance. 2 In addition, the resistance to cultural change by contractor and government 
as customer has not been an enabler for acquisition reform. 

The impact of acquisition reform extends beyond the procurement of defense systems and 
related technology. Given the relationship of the military industrial base to the national 
economy and world peace, an acquisition eform strategy that effectively and efficiently 
supports both the industrial base and our war-fighters is needed. At risk is our ability to 
control our future, to maintain our way of life, to produce wealth, and to continue to 
contribute to the advancement of humanity. 

Acquisition reform has evolved to the level where the most effective relationship between 
technology maturation and product procurement is demanded. Several models of such a 


1 Eugene Gholz, and Harvey M. Sapotsky, "Restructuring the U.S. Defense Industry.* International Security 
Vol. 24, No.3 ( Winter 1999/2000), p. 8. 

1 Charles H. Fine, Clockspeed: Winning Industry Control in the Age of Temporary Advantage^ Reading, 
MA: Perseus Books, 1998), pp, 116-124. 
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relationship exist. Some are commercially derived while others are defense unique. 3 
Factors such as technology and product complexity are important determinants in the 
successful execution of many of the models. Existing processes, practices, and statures 
may be barriers and or enablers in implementing these models. My objective today is not 
to critique the various models. Rather, based on my economic incentives research and my 
NASA experience, 1 will give my views on the critical relationships between contractor 
and customer that enable the maturation of technology and the procurement of defense 
systems. 

2.0 Economically Incentivized Production 

To develop technology to a high readiness level and to procure products in production 
require the establishment of an environment that facilitates a "win-win" solution for 
government and contractor. Several factors contribute to the establishment of this desired 
environment. The primary factors enabling a "win-win'' solution are: 

• Firm, understood, agreed upon requirements 

• Exposed, clearly understood, acceptable risks 

• Sufficient "top cover" 

• Effective leadership and management 

• Institutional commitment to a successful program 

• Agreed upon performance metrics 

• Effective use of properly skilled joint government-contractor IFTs 

A critical assumption in identifying the necessary conditions above is that the program 
has funding to allow continuation to the next stage of development including the insertion 
of technology developed in parallel to, or outside of, product acquisition. The existence 
of these factors in successful "win-win" incentivized contracts has been documented in 
several case studies supporting my research and in NASA technology development 
programs. The case studies include the Sensor Fused Weapon (SFW), Joint Direct Attack 
Munitions (JDAM), C-17, C130J, F-414 engine, and F-117 engine. The NASA advanced 
technology development programs include the High Speed Research Program (HSRP), 
the Advanced Subsonic Technology Program (ASTP), and the joint NASA - DoD 
Integrated High Performance Turbine Engine Technology Program (IHPTET). 

In each of these programs government goals, contractor goals, and joint government- 
contractor goals are identified and accepted by both parties. Government goals include 
affordable defense systems with performance characteristics exceeding known and 
anticipated threats and technology developed to a high readiness level. Contractor goals 
include customer satisfaction, planning stability, good financial performance, and cash 
flow. A joint goal common to both government and contractor is to assure the pre- 
eminent military strength of our country by providing the war-fighter with superior 
weapon systems. 


3 See, for exampfe, Alton D. Slay, Wesley L. Harris, et. al.. Defense Manufacturing in 2010 and Beyond: 
Meeting the Changing Needs of National Defense (Washington, DC: National Academy Press, 1999), pp. 
62-72. 
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Strategic government goals include the continued existence of a robust and competitive 
defense industrial base. Also strategic contractor goals include enhanced reputation and 
continuous improvement of its processes and practices to remain competitive. 4 

My research has revealed that the pillars supporting the desired enabling environment are 
(1) mutual trust and (2) mutual respect between government and contractor. Mutual trust 
and mutual respect eliminate the traditional challenge associated with the Principle-Agent 
model of contracting in which there is an asymmetry of information between the two 
parties. This mutually trusting environment leads to a open flow of information between 
parties. Mutual trust and mutual respect are the sine qua non of today's acquisition 
reform. This conclusion is supported by my research and is valid for both technology 
development and product procurement. 


2.1 Outcomes of "Win-Win" Solutions 


Both government and contractor receive gains in a "win-win" partnership. Government 
gains include: 

• Technically sound systems 

• Reduced cost 

• Most competitive product 

• More complete understanding of contractor goals and constraints 

• Potential for additional cost reduction 

Contractor gains are: 

• Reasonable-firm government commitment 

• Reward for accepting risk 

• Enhanced corporate reputation 

• Government assistance in becoming more lean 

• Share in cost reduction savings 


With this set of outcomes, the interests and welfare of the U. S. tax payer and the 
corporate shock holder are protected. The goals of the government and contractor as well 
as the joint common goals have been achieved. The near term and long term interests of 
the nation are also protected through a "win-win" partnership of government and 
contractor in the development of advanced technology and acquisition of high 
performance weapon systems. 


4 J. C. Kreidel, "DoD, Industry, MIT Set Sights on Ensuring Military Might Economic Incentives lor 
Systems in Production", Program Manager Vol. XXIX, No. 1 (January-Febntary 2000), pp. 72-75. 
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2.2 Lessons Learned 


The primary lessons learned through my investigations and study of successful programs 

covering a very diverse range of technological complexity are: 

• High level senior/ranking commitment and support enhance program success. 

• Information and risk, openly shared, precede development of economic incentives 
through delicate negotiations. 

• Firm government commitment, over a finite time period, to the program reduces 
mutual risks. 

• Contractor investments of its resources to reduce cost enhance program success. 

• Innovative use by government of the following concepts can form a foundation for 
risk-reward balance: 

• Multi-year contracting 

• Waiver of Certified Cost and Pricing Data (CCPD) requirements 

• Performance based payments (PBP) 

• Economic Order Quantity (EOQ) funding 

• Joint Cost Modeling 

• Variations in Quantity (VIQ) options 

• Leadership and use of joint (government and contractor) IPTs increase 
communication and information flow. 

• Mutual trust and mutual respect enable internalization of strategic goals and visions. 


The above lessons learned are relevant to the issues being addressed by this 
Subcommittee. I propose to further demonstrate that relevance by arguing that by 
utilizing appropriate incentives, technology can be developed in parallel to the acquisition 
of a product which depends upon that technology in order to satisfy performance 
requirements. 


3.0 Incentivized Parallel Technology Development 

A few comments on successful development of critical, complex technology in advance 
of full system acquisition. In the early 1990's, NASA Office of Aeronautics under my 
leadership as Associate Administrator for Aeronautics did develop and manage two 
technology development programs. These programs were the High Speed Research 
Program (HSRP) and the Advanced Subsonic Technology Program (ASTP). The NASA 
Office of Aeronautics also worked jointly with DoD and industry to develop advanced 
gas turbine components within the Integrated High Performance Turbine Engine 
Technology Program (IHPTET). These technology development programs were 
successful due to several factors, namely: 
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• Stable, government-contractor jointly developed subsystem requirements 

• Sharing information on risk 

• Identification of participant specific goals and common goals 

• Mutual trust and mutual respect between government and contractor 

• Existence of potential markets for commercial technology and next generation 
weapon systems for military related technology 

• Stable funding 

• Firm government commitment 

• Effective, lean leadership representing government and contractor 

• Exploitation of technology developed offshore 

• Economic threat and military threats 


Note the very strong commonality between the factors leading to successful development 
of technology parallel to system acquisition and the factors that enable a "win-win" 
solution for programs in production. At the most fundamental level, the environment for 
favorable development of advanced technology is very similar to the environment for 
acquisition of defense systems in production where the technology risk is low 
corresponding to technology at a high maturity level. 

Advanced technology at a high maturity level is essential to the acquisition of affordable 
weapon systems with requirements for superior performance. Without advanced 
technology one is not able to develop superior performance systems. Without this 
advanced technology at a high maturity level, acquisition policy and practice is null since 
there is no worthy objective or defense system to procure. The importance of advanced 
technology at a high maturity level is so great that the government must incentivize the 
contractor to develop advanced technology. The government as customer must place a 
premium on the development of advanced technology to a high maturity level. The 
premium must compare favorably with the profits and or awards, both financial and 
non-financial, that a contractor would earn in producing systems for acquisition. 

The necessary high premium will drive the government to a lean portfolio management 
condition within the constraints of at least a fixed research and development plus 
acquisition budget. Advanced technologies selected for development to a high maturity 
level must be based on realistic projections of need. Otherwise, the development of 
advanced technology would become a sandbox for non-value added play. Without a 
realistic projection of need, the probability of inserting matured advanced technology into 
a defense system would be low. Traditional technology roadmaps that are produced sans 
the rigor and quantitative refinement necessary to increase the probability of later 
inserting developed technology from that roadmap will not provide sufficient incentive to 
the contractor to accept the risk associated with developing such technology to maturity. 
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An improve chance of success in developing advanced technology to a high level of 
maturity would be enabled by the adoption of a common, quantitative-based language 
and assessment tools. Specifically, qualitative descriptions of technology readiness levels 
are insufficient and inconsistent with the above mentioned realistic projections of need. 3 
The desired quantitative-based language and assessment tools are recommended to be 
derived from probability theory . Thus, each technology readiness level would be 
defined by probability bands . Government as customer and contractor would have a 
quantitative, well-defined target at which to direct their efforts to develop advanced 
technology. Obviously, requirement instability limits the effectiveness and utility of a 
quantitative-based set of tools and languages defining technology readiness levels. 

4.0 Conclusion 


In conclusion, I would like to emphasize the complex and interlocking structure of our 
acquisition policy and practices - a necessity in the complexity of major shifts in our 
defense industry base, declining acquisition budgets, and evolving threat characteristics. 
There exist case studies of successful parallel development of advanced technology to 
high maturity levels where the government is customer. Likewise, there exist several case 
studies of successful acquisition of defense systems in production. In very important 
areas, the processes of both successes are quite similar. This similarity should be 
exploited to build creative structures to ensure the development of advanced technology 
to high maturity levels essential to the acquisition of affordable weapon systems with 
superior performance. 


Wesley L. Harris 

Professor of Aeronautics and Astronautics 
Director, Lean Sustainment Initiative 
Department of Aeronautics and Astronautics 
Massachusetts Institute of Technology 
Room 33-410 
77 Massachusetts Avenue 
Cambridge, MA 02139 - 4307 
TEL: 617-253-0911 
FAX: 617-2.58-0863 


5 See, for example, "5000.2 FINAL Coordination Draft", Department of Defense, April 2000, DoD 5000.2- 
R, Appendix 6, Technology Readiness Levels and Their Definitions, pp. 109-110. 
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Mr. Shays. Thank you. There will be questions. And I’ll just 
throw it out to all of you. 

First off, you were all here for the entire hearing, and I thank 
you for that, because the value is that you can comment without 
me having to repeat the questions. 

Conceptually, a “best practice” model for acquisition that makes 
sense in the private sector for private development also makes 
sense for the private sector for a public customer. Would you all 
agree with that? 

And if not, if you would, qualify it where you would — I mean con- 
ceptually. I make the assumption nobody would disagree. 

Mr. McNaugher. You’re saying that the commercial best prac- 
tices is transferable to the public process of 

Mr. Shays. Yes, and that we should seek to use commercial best 
practices. 

Mr. McNaugher. The only caveat I would raise, Mr. Chairman, 
I mean, obviously no commercial firm operates in the kind of politi- 
cal environment 

Mr. Shays. Just a little louder. I know you’re not feeling well, 
and I said I wasn’t sympathetic; but I didn’t want to give you sym- 
pathy because I didn’t want it to get worse. 

Mr. McNaugher. There’s a lot of differences, obviously, between 
buying weapons as a public good and developing a private sector 
item. The only difference, though, that leads me to question the 
use, the extensive use of best practice models is I don’t think in the 
commercial world very often you systematically take these huge 
leaps out into the unknown, as we did during the cold war and, I 
think, still do; that is, not only the components of a system, but the 
integration of a system really represents a dramatic jump. 

Mr. Shays. But you don’t have any fear that we’re doing exten- 
sive use of this now. I mean, we’re not, this is the — I’m not aware 
that a commercial best practices model is running rampant 
through our government. 

Mr. McNaugher. No. What I’m saying is, trying to apply it to 
the government, to this process, has limits. Most commercial firms 
insofar as they’re doing marginal improvements are aware of risks 
and costs at a fairly good level of certainty. You know, most of the 
discussions of risks in the early phases of cold war development 
programs proved in the long run to have been wrong. You don’t 
know what the numbers really are. You have an idea of a cost per- 
formance curve 

Mr. Shays. And they were almost always understated. 

Mr. McNaugher. Yes. That’s not an indictment of anybody’s be- 
havior. Technology just doesn’t give up its secrets very easily. So 
we have these very precise discussions during “dem-val,” but, the 
numbers and curves are going to shift. That’s again why you need 
more flexibility down the road. 

I think commercial practice may not have to deal with that kind 
of uncertainty. 

Mr. Shays. OK. We’ll touch on that a little bit more in a second. 

Mr. Larkins, when I think of 3M, I would think that you use a 
knowledge-based process, a best practices model for acquisition as 
defined by GAO when they talked about knowledge, point one, 
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match is made between the customer’s requirements and the avail- 
able technology. 

Mr. Larkins. Yes, sir. 

Mr. Shays. Knowledge, two, when the product design is deter- 
mined to be capable of meeting performance requirements. And 
three, knowledge point three, when the product is determined to be 
producible with cost schedule and quality targets. I would think 
3M would just be right in the center of that kind of philosophy. 

Mr. Larkins. Yes, sir, that’s correct. 

Mr. Shays. Where it might differ is that I think of 3M, with no 
disrespect, but just in terms of putting it in perspective, that you 
are seeing customer needs and then you’re seeing how you can 
meet those needs with products that you have potentially available 
in technology that you have available. 

I make an assumption in most cases that technology is pretty de- 
veloped, or sometimes do you have to — do you see a need and you 
just start out from scratch with a technology? 

Mr. Larkins. That happens, sir. 

Mr. Shays. What happens? 

Mr. Larkins. Where we will, in fact, start from scratch and de- 
velop the technology. A lot of it depends on how broad your window 
of opportunity is in the market. 

What we are finding today is that if you don’t meet a market 
need quickly, then the market need is met by someone else. So 
what we try to do is do a rapid technology assessment, determine 
whether in fact we have technologies in house to meet that need. 
If we don’t, then we will go out and make a technology partnership 
or acquire a technology. 

Mr. Shays. And admittedly the — so time is of the essence? 

Mr. Larkins. Absolutely. 

Mr. Shays. But your technology requirements may not and cer- 
tainly wouldn’t be as complex as developing an aircraft or 

Mr. Larkins. That’s very fair to say. 

Mr. Shays. So we need to have some empathy for the task at 
hand, that applying best commercial practices is a very important 
effort, but certainly we can carry the commercial to the defense; 
and we may be — it may be unfair to say that you — DOD needs to 
do it just like we do it in the private sector. 

Mr. Larkins. Yes, I agree that it would be unfair to say that 
their technology integration problems are as easy as ours, because 
they certainly are not. And — but I would like to agree with my col- 
league, Dr. Harris. 

I think one of the reasons we are successful and are able to 
achieve the type of record in product integration scale-up that we 
do is because we do have a significant investment in technology. 
We have a broad base of technology platforms that we can tap into, 
and they’re ready to be integrated. 

Mr. Shays. But it would be unlikely for 3M to go into develop- 
ment and production without the technology there to back it up? 

Mr. Larkins. We absolutely would not. 

Mr. Shays. You would not. So the general principle that there is 
significant logic in developing the technology before you went into 
development and production still holds for the defense as well. 
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Mr. Larkins. Well, all I can say is that we do not go into a prod- 
uct scale-up mode until we have the technologies in hand. 

Mr. Shays. So, then, if you can’t meet the deadline, you just 
don’t produce the product? 

Mr. Larkins. That’s correct. 

Mr. Shays. Dr. Harris, do you want to comment on the questions 
I have asked so far? 

Mr. Harris. Yes, I would like to comment, sir. 

The use of the words “commercial best practices” and our consid- 
eration of the — this environment of acquisition of defense systems, 
that phrase is a very loaded one. It has many meanings and usu- 
ally it differs, depending upon the speaker. 

Mr. Shays. Is that — if I could interrupt — is that because there is 
no, one, so-called “commercial best practice”; there are — different 
companies have their version of best practices? 

Mr. Harris. Well, that’s somewhat it, sir, but I think — my state- 
ment is based primarily on the fact that there’s no commonly 
agreed-upon body of knowledge as to what it really is. It’s a sort 
of catchall phrase that has caught many up into some more politi- 
cal stance than real substance. 

For example, the question of scaling along technology complexity, 
the question of scaling in terms of number of products, the ques- 
tions of availability of markets all would impact commercial prac- 
tices. And you don’t have a serious discussion of commercial best 
practices being transferable along what is in fact scalable. 

So I’m somewhat personally concerned as to whether — what we 
really mean when we say that DOD or the government ought to 
rush immediately into commercial practices. It just doesn’t have 
the substance that I think it must have in order to hold those who 
practice within government accountable when we say that they 
ought to use or should be using best practices. 

Mr. Shays. So you would be more sympathetic to the view that — 
that when we use TRL, technical readiness levels, that it be more 
a tool than a requirement. And you heard them refer 

Mr. Harris. Yes, I did hear that discussion, sir. Yes, Mr. Chair- 
man. I think we’re wrapping ourselves around an axle with the 
qualitative discussions of technology readiness levels. Until we can 
move to a position where we can quantify what we mean by tech- 
nology readiness, we can’t hold those who practice, who acquire the 
weapons systems, accountable. Because they and their contractors 
will always be at each other’s throats as to what they mean when 
they present an argument or position on technology readiness lev- 
els. 

So I’m calling for more quantitative language and assessment 
tools to enable all of us to understand precisely where we are with- 
in this arena called “technology readiness levels.” 

Mr. Shays. That’s spoken like a true academician. You’re fulfill- 
ing your role perfectly. 

Well, one of the things is — and I’m not burdened by being a law- 
yer, and I don’t have any perceived — and the reason why I love this 
committee and love the work as a legislator, I’m not trying to prove 
a point. I’m trying to understand what we as a committee need to 
recommend to the full Congress and to the executive branch. 
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But intuitively, I have accepted as a fact — and, Dr. McNaugher, 
I’m going to try to have you separate a little bit the so-called poli- 
tics side of this, and then we’ll get back into that. But I accept as 
just very logical, without a lot of empirical knowledge, but just your 
own life experiences, that if you are going to be developing tech- 
nology and development, and fail to come to grips with some tech- 
nology; and then you’ve got a point where the technology hasn’t 
been resolved and yet you’re still moving things along, that your 
costs are going to go up significantly. You’re going to begin to slow 
down your development after you started, particularly in produc- 
tion. 

And I accept the point that you made, Dr. McNaugher, that 
sometimes we haven’t perfected the product until we have produced 
100 planes. And that’s a real negative. 

So, Dr. Harris, tell me why intuitively I shouldn’t accept the fact 
that we should develop the technology before we go into develop- 
ment, but particularly into production. 

Mr. Harris. Well, I can’t tell you that. I want my testimony to 
read very clearly that we 

Mr. Shays. Don’t put words in your mouth. 

Mr. Harris. We as a Nation have, in my opinion, been successful 
in developing technology in advance of acquisition of the full sys- 
tem. My experiences at NASA with the high-speed civil transport, 
the new 700,000-pound Mach 2.5 airplane, we developed that tech- 
nology with industry as partners. The airplane was never built be- 
cause it was a business decision, not because the technology was 
not there, or was designed to be there, in advance of building the 
airplane. So we know how to build or how to develop technology in 
advance of acquisition. 

Mr. Shays. But the question is, is it going to be cost effective? 

Mr. Harris. Oh, I would say it definitely is cost effective. And 
the effectiveness goes up, I believe, with the complexity of the final 
product. If you’re talking about very complicated systems that re- 
quire a real stretch in technology, then clearly the effectiveness is 
going to be there, the cost effectiveness. 

Mr. Shays. I’ll allow the other two to jump in in a second, but 
where I’m not with you is, I would think that waiting to develop 
the technology before you did the development, but particularly 
production, would be to your advantage as long as you don’t have 
the time restraint of having to get into the marketplace because 
someone else is going to build the product. 

Mr. Harris. Well, I agree, as I know the threat, at least the mili- 
tary threat, that this Nation currently faces, we have that time. 

Mr. Shays. Right. OK. 

Any other responses to any points that either of you, Mr. Larkins 
or Dr. McNaugher, any comments you want to make? 

When you heard the debate between — the dialog between the 
first panel and the second panel, I want each of you to tell me your 
reaction not in — as concise as you can; and tell me how you reacted 
to the two different testimonies, because I think both testimonies 
were quite excellent. I think GAO gave us a wonderful vehicle in 
which to have this debate and dialog. 

But I would love to know how you reacted as you heard it. 

No criticism of the committee allowed, just the other side. 
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Mr. Harris. Well, I’ll go first. 

Mr. McNaugher. Yeah, feel free. I’m happy to defer. 

Mr. Harris. I thought the first panel was again a very excellent 
presentation. I think there was an honest mistake in interpreting 
what technology readiness levels were meant. Was it to be on a 
full-up system or a comparable system? And that led, I think, to 
a downgrading of the so-called “technology readiness levels.” 

What I thought was missing in the first presentation was a nor- 
malization of the two possible streams to the same system, one 
stream as to what we currently have, or at least what the first 
panel proposes that we have, that we buy this complex weapons 
system and develop the technology while we buy it. Another stream 
would be to develop the critical technology off stream, then insert 
it when we are ready to move forward with the full system. 

Now, my question is this: What is the cost going along the first 
stream and what is the total cost going along the second stream? 
And what are the time lines involved? When does the clock start? 

Mr. Shays. How would we know that if we don’t know if we can 
actually develop the technology? 

Mr. Harris. As far as I know, sir, no one has ever done a com- 
parative analysis of that sort — unless Dr. McNaugher has done it. 
Yet we get this constant bombardment of how to develop systems 
and how to develop technologies that support systems off line. 

But the answer is, Mr. Chairman, no one really knows. 

Mr. Shays. What I heard, and then I’ll open it to the other two 
of you, when I found myself wondering if moving forward with de- 
velopment represents — where you still have high risk on tech- 
nology, whether that represents a significant negative. But I clear- 
ly feel that if you went into production without the technology, 
then we are really opening up potential costs — I mean, that ratio. 
So I found myself more tolerant of DOD kind of wanting to go to 
that development stage. 

I would become very concerned if they wanted to go that one step 
further. In other words, I think there’s still this high risk in some 
of their technology. 

Mr. Harris. OK. I don’t know the specific technologies involved 
with this weapons system, as I am not a consultant to any of the 
primes. But there are two different ways of getting to the same end 
goal; a highly complex system is what we don’t have a comparative 
analysis to. 

Mr. Shays. Fair enough. 

Dr. McNaugher. 

Mr. McNaugher. First of all, I don’t know enough about the 
TRL methodology to comment on it explicitly. I think it’s absolutely 
appropriate to be hammering on the readiness of these tech- 
nologies. The disagreement between the two prior panels — and not 
everybody agreed that there was a disagreement — seemed to circle 
around the question of whether you’re assessing risk in the context 
of systems integration or the risk of specific technologies. I can at 
least understand why you get a different number in each of those 
cases, given what I said earlier. 

And so the one upshot I would have, and I think I’m echoing 
something Dr. Harris said, is whenever you move or this project 
moves from “dem-val” to EMD, don’t think that just because you 
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have reached TRL 7 for components, you’re out of the woods. 
There’s a lot of development work and there’s going to be a lot of 
sometimes nasty surprises up ahead. 

Mr. Shays. And I don’t think GAO was suggesting that. 

Mr. McNaugher. Agreed. So I can understand why there might 
be a different measure, depending upon whether you’re thinking of 
the specific technology or that technology in the context of an over- 
all system, because the system does pose its own uncertainties. 

Mr. Shays. Mr. Larkins. 

Mr. Larkins. There seemed to me to be two basic issues that 
came up. I think 

Mr. Shays. Can you move the mic just a little closer. 

Mr. Larkins. There seem to be two basic issues of disagreement. 
I think there was an awful lot of agreement, but there were two 
basic issues that I got that there was disagreement. One was 
that — one was the TRL as — the definition used in each individual 
category. And again I have to agree with Dr. Harris here; I think 
that if you’re going to use categories to evaluate a project, you 
should have general agreement by everybody, the people who are 
being evaluated and the evaluators, as to what those categories are 
and where they fit. And there obviously wasn’t any there. 

The other disagreement was that there is a substantial difference 
between a fully integrated system with eight apparent critical tech- 
nologies versus evaluating each of the individual critical tech- 
nologies. 

And frankly, at 3M, when we look at integrating several new 
technologies into a program, if we consider, for example, that each 
technology has a 90 percent chance of success, then you have to do 
a multiple of those chances of success to arrive at a final chance 
of success for your program. So if each of your programs has a 2 
percent chance of failure and you have — or each of your tech- 
nologies has a 2 percent chance of failure and there are four tech- 
nologies, you’re looking at a 16 percent chance of failure right 
there. 

So I am completely sympathetic with what the Department of 
Defense is saying here. 

Mr. Shays. And yet you would be sympathetic with the fact this 
integration sometimes may mean that the technology doesn’t work? 

Mr. Larkins. That’s correct. And that’s why we frankly take a 
very conservative view of that approach. We — and believe me, we 
do take risks, but they are calculated risks; and the greater the 
risk on a commercialization program, the tighter we monitor it. 

Mr. Shays. Do you have any questions? 

Mr. Chase would like to ask questions. 

Mr. Chase. Mr. Larkins, what kind of incentives do you provide 
your teams, your project teams, to ensure that a new product that 
you’re trying to develop comes in on time and within budget? 

Mr. Larkins. Well, there are, in my view anyway, three different 
types of incentives, peer recognition and reward — and we have pro- 
grams in place which recognize individuals and teams for outstand- 
ing success in product introduction — and, of course, promotion. And 
when a person is a program manager and they’re involved in a suc- 
cessful product introduction, there are certainly promotions that 
are involved in that type of an activity. And finally, financial incen- 
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tives as well, which go along with the promotion and separate from 
that. 

So recognition and reward and promotion. 

Mr. Shays. Thank you. I was going to ask that question after you 
did. Because I just wonder if in the — the Federal Government, 
when they are utilizing the services of a private company, whether 
those companies themselves have their own internal programs to 
encourage as much innovation and extra effort and so on. 

Let me say this to you. I’ve really asked the questions I wanted 
to ask. I’m going to allow all of you to make some closing comments 
or ask yourself a question you wish I had asked, that you think I 
should have asked. 

I would also like to just ask if Mr. Rodrigues would mind coming 
to the corner here, just to kind of share your observations. 

Is there anyone from DOD that was here that would like to or — 
if they hear a comment — do we have a representative from DOD 
who would be? 

We had let them know they would be invited if they were able 
to stay. I realize they couldn’t necessarily stay. If you could, just 
sit on the corner here, if you could do that. 

Are there any comments that you would like to make or observa- 
tions? If so, I would move the mic over to you. 

This is not — the purpose of this is not a debate, but just to say 
things that you would want us to focus in on. 

Mr. Rodrigues. I guess I would like to say a couple of things to 
try to add a little clarification. 

The work that we did on the Joint Strike Fighter and the appli- 
cation of the TRLs, being able to do that was the culmination of 
years of work. It was not easy to define a model that reflected what 
we defined as, at this point, “commercial best practices” that can 
be applied in DOD. That was gleaned from a lot of work with a lot 
of companies, and working with the Department looking at suc- 
cesses and failures and trying to categorize what people do to come 
up with knowledge points that are critical to being able to move 
forward successfully. 

When we put that over at the Department of Defense for com- 
ment, the original comments that we were getting back were very, 
very negative because it would require a significant change in the 
way they do things. 

Dr. Gansler, coming on board, he took that on and rewrote the 
comments; and I can tell you I never get comments like this. At 
the end, he writes a personal note, on the formal comments coming 
over, “A very thoughtful and helpful report, Jack.” Believe me, I 
never get these. 

What we then did is, we said, OK, if you agree with — if you could 
put that chart up, please. 

Mr. Shays. You know, it’s interesting, just a nice little comment 
like that goes a long way. We all need to do it, all of us. 

Mr. Rodrigues. That’s the only one I’ve ever gotten. 

So if you look at this chart, then we said, OK, if everybody can 
agree with this — and we had to come up with a model, run it back 
by all the commercial companies we did, because they didn’t know 
how to define what they did. 
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They don’t think about it in this term. Nobody had a model for 
us to use. We had to work and find it. Find it, define it, get it un- 
derstood by both parties, DOD and the commercial guys; and let’s 
agree to some things. 

Once we got agreement on that, we went back and said, if this 
is true, let’s peel the onion. If technology development, separating 
technology development from product development is so critical, 
how do you do it? 

Mr. Shays. I need to know what your bottom line is. 

Mr. Rodrigues. My bottom line is, we did a whole study to come 
up with the technology readiness levels. We didn’t get there by ac- 
cident. We did a whole thing focusing on knowledge point one. And 
we worked with a lot of individual technologies, both commercial 
and defense. What we found was that they work, that they can pre- 
dict outcome, that if you meet the right technology level, you can 
get success. If you don’t, you are virtually guaranteed failure. 

Mr. Shays. You’re a brave guy to say that right next to Dr. Har- 
ris. 

Mr. Rodrigues. I’m not guaranteeing there aren’t better ways to 
do it. I’m just saying they are a great indicator. 

Now, where I was going with this was or trying 

Mr. Shays. Just don’t get too deep here. 

Mr. Rodrigues. In that report, before we applied TRLs to the 
JSF Program — see, we applied it in the abstract and didn’t criticize 
anything — we never said anything. 

We said, here they are. Here’s a tool. Isn’t it wonderful? Can’t 
you guys use this? Doesn’t it make sense? When we defined it in 
there — and this went to the Department for comment, and I’ll read 
you their comments. We said — we defined TRL 7. We were talking 
about where does integration fit. There are two types of integra- 
tion, there is subsystem integration and there is product integra- 
tion. 

Mr. Shays. OK. 

Mr. Rodrigues. We got a whole bunch of technologies that are 
at subsystems. Integrating proven technologies into a product is a 
real challenge, and it is something that has to be done. Matching 
technology to requirement up front doesn’t stop that from happen- 
ing. You still have to do it. It’s a challenge. 

What we’re trying to do is isolate that, because what should be 
done in product development — isolate that from the technology de- 
velopment which is proving the technologies in the subsystem 
forms in which they have to be, so you’re not trying to do that con- 
currently, especially on pacing items. So we said when radio com- 
ponents — this is in the TRL, report — are assembled inside a case 
that resembles a final radio design and then demonstrated aboard 
a surrogate of the intended aircraft, the radio reads TRL 7 now 
when they comment on this report, they say that the Department 
agrees — this is their writing, it’s not mine, reprinted in the re- 
port — the Department agrees that TRL is an important input and 
is necessary, but adds they’re not sufficient alone to decide when 
and where to insert new technologies in a weapons system pro- 
grams. 

Military system development decisions require a total ownership 
cost approach through the entire life cycle system. Now I totally 
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agree with that. We weren’t trying to imply that once you reach 7 
you have to put it on a product. We were saying if you don’t reach 
7, you’ve got problems and you shouldn’t put it on a product. They 
went on to say in their comments 

Mr. Shays. I want you to bring this to a close. 

Mr. Rodrigues. This is their comment. The Department concurs 
with GAO that the weapons system program manager should as- 
sure that technology is matured to TRL 7 — I just read how we de- 
fined it in the report and they agreed — before insertion into a new 
system. They agreed with that here. 

The difference in the JSF study was we then cross-walked that, 
and said, hey, you know what? When you apply this on the pro- 
gram you’re doing, you’re not there and you shouldn’t move ahead. 
It wasn’t until we made that cross-walk in a draft report for com- 
ment that all of a sudden this issue of a misunderstanding, came 
up. It came up once they saw how it was used. If they had read 
this report in detail, which I had provided to them, they would 
have seen how it would be used. 

Mr. Shays. Let me just say I think they did read the report. The 
advantage that GAO or Members of Congress have is that we can 
look at something that someone else is building or doing, and we 
can sit with hindsight and all the other things, and really be pretty 
analytical about it. 

I think your report was done very tastefully. I think it served a 
tremendous use here and will be used by other committees because, 
as you know, your report was used by the Armed Services Commit- 
tee even before we released it today. 

I think DOD was fairly respectful of your analysis. I think there 
are — I have a sense of where our disagreements are. And a lot of 
good is going to come from the report. And — but I’m excited that 
this is a program being used by Defense. I want to make sure that 
they’re using it to the fullest extent possible. 

Let me just — I’ll let you say another comment here, but let me 
just ask Dr. Harris or Dr. Larkins or Dr. McNaugher: Any question 
you want to ask yourself and then answer brilliantly? 

Mr. Harris. Mr. Chairman, no question, but I would like to con- 
clude by simply repeating myself. 

We, as a Nation, have demonstrated the capability to produce 
weapons systems driven by advanced technology. To become more 
efficient at the acquisition process, we must continue to develop ad- 
vanced technology in parallel to acquisition of full, complete sys- 
tems. Key elements in this efficiency are, No. 1, incentivize indus- 
try with favorable premiums, develop this advanced technology; 
No. 2, realistic projections of advanced technology needs, meaning 
a management of our DOD technology advancement portfolio; 
third, use of quantitative-based language and assessment tools. 

Mr. Shays. Thank you very much. 

Mr. McNaugher. Three points, the first taking off from Dr. Har- 
ris’ last remark; and it’s a subject of a different hearing, but one 
that should be kept in mind. 

We have never found how to make R&D per se profitable in the 
defense industry. The historic tendency is, firms sink their own 
money into development and get well in production. If you want to 
relax this process and maybe even cancel the occasional system, 
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you probably want to have a way, a mechanism, for making R&D 
profitable. We’ve never found that, and that’s the way you encour- 
age industry. It’s a profound problem in the defense industrial 
base, precisely in this area we’re moving into. 

My second point has to do with the earlier discussion of the ten- 
ure of program managers. I must say I’m fascinated with the idea 
of a program manager who actually has to live with the results of 
his or her decisions 5 or 6 years later, I think it would be a pro- 
found change. Again, though, stepping back and looking at the full 
sweep of American history, remember that before World War II the 
Army had an Ordnance Department which was a bunch of full-time 
program managers and technocrats, if you will. They wore uni- 
forms, but they didn’t rotate. 

I think the Army’s conclusion, as an institution coming out of 
World War II, was that the Ordnance Department was not atten- 
tive enough to the user, the actual operator. And it was more fix- 
ated on the technologies than on military capability. What the 
Army did in the 1950’s and 1960’s was begin to substitute line 
users as program managers, and but also as the chief of R&D, for 
example. 

Mr. Shays. Wasn’t there also another danger that they developed 
too cozy a relationship with the organization they were buying 
from? 

Mr. McNaugher. Well 

Mr. Shays. Over time you just began to become almost their ad- 
vocate, not necessarily their 

Mr. McNaugher. My sense of the ordnance department, to the 
extent that I understand the history, is that it had a somewhat 
thorny relationship with commercial firms because it was doing its 
own R&D, and it was always being besieged by Members of Con- 
gress pushing Samuel Colt’s pistol or rifle. So it was a more com- 
bative relationship. 

Mr. Shays. So — I didn’t realize, they did the R&D; I just thought 
they were 

Mr. McNaugher. Springfield Arsenal. The arsenals did R&D, so 
they were competing with industry to some extent. But the point 
is, a program manager tenured for 15 years or 10 years may be- 
come very attentive to sort of the permanent features in the envi- 
ronment, of which the political structure is one, and not as atten- 
tive to the military user. So maybe there’s a length of tenure that 
sort of balances those needs. I don’t think it’s 3 years either. I 
think it’s longer than that. 

And finally, at the risk of complicating things, I would just sepa- 
rate technologies into two bundles, slow-moving and fast-moving 
technologies. If you look at an airplane, the engine airframe com- 
bination had its fast-moving days back in World War I and the 
1950’s, and that’s slow-moving technology. You can afford to stop, 
assess that technology, test it. 

If you look at the electronics in the avionics 

Mr. Shays. Let me just interrupt you a second so we can ask a 
question of the recorder. 

Thank you. 

Mr. McNaugher. If we look at avionics in the cockpit of that air- 
plane, which may come to 40 or 45 percent of the cost of the air- 
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plane, we’re looking at technologies where if you stop and test and 
decide, you know, they’re already obsolete — I mean, they’re turning 
over in a year, 18 months, 2 years — and I’ve always felt that we 
really need — now that electronics technologies are seen to be the 
key to the revolution in military affairs we really need to handle 
these by different commercial practices. One can be much slower 
and more judicious because you’re way out on the flat of the cost 
performance curve; you’re pushing for that marginal improvement 
in jet engines or airframes. The other is just so fast-moving that 
you almost have to be turning your design and your tests over con- 
stantly. The DOD needs a different approach entirely. 

Mr. Shays. I thank you. 

Mr. Larkins, I knew that would bring you out. 

Mr. Larkins. If I could just make one. 

Mr. Shays. Would you move the mic, please, toward you. 

Mr. Larkins. If I could just make one additional comment, I 
would say that our experience in industry with the importance of 
focusing on fewer larger programs, that having trained people who 
are — whom we call program managers or program development 
managers, whose expertise is scaling up new programs. We are be- 
ginning to see within industry a focus on that as a discipline. 

3M traditionally has what we call a dual-ladder system where we 
have technology people on one side of the promotion scale and man- 
agement people on the other side of the professional scale. So you 
have research and management moving up together, the same op- 
portunities for advancement, the same opportunities for pay, this 
kind of thing. But we are now in industry beginning to see a third 
leg or a third ladder in this process, which is a focus on program 
management. 

Mr. Shays. Any other comment? 

Mr. Rodrigues. I would like to make one, because this becomes 
a real area of concern; it is related to this. I have started looking 
at some data really hard-looking at this whole issue, this issue of 
profitability for research and development. In private industry, re- 
search and development is investment, the return is on sales. You 
don’t make money in R&D. You make money building a product 
and selling it to somebody. The same principles should apply in the 
Department of Defense and should apply in the defense industry. 
The reward for doing the right thing is you sell us something that 
you make a profit on. 

When I looked at this issue and was trying to study this, what 
I find is when I look at tactical aircraft, from 1973 to 1991 we built 
anywhere, annually, from 350 to 500 aircraft. From 1991 to today, 
we’re building handsful, 40, 50 aircraft, sometimes much, much 
less than that. 

And then we have an industry saying, we’re not profitable; Wall 
Street is beating us up. There are a lot of other factors, but one 
of them is, we’re not producing a lot. 

We got into a cycle where we’re heavily into R&D and so now the 
issue becomes, how do we make R&D profitable? The issue be- 
comes, we have a modernization strategy — I think this is a real 
issue that we need to take a hard look at — that drives us to cycles 
that make the industry unprofitable at times. 
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Is the solution to pay them for R&D, pay profit rates, increase 
profit rates on something that is normally an investment account; 
or is the solution to better plan on modernization so that you 
equalize production, so people can make money and we can mod- 
ernize? 

Mr. Shays. I think it’s a very fine point. 

I saw Dr. Harris start to waiver a little bit here. I am at over- 
load. So whatever you guys are, I’m at overload; I see a new re- 
corder, and we’re done. 

Julie Thomas, thank you for all your work. There’s no question 
you were working hard today. 

And I enjoyed this hearing. I learned a lot. I think our witnesses 
and all three panels were terrific. And I also appreciate the staff 
for both the majority and minority for all their good work. So, ev- 
eryone, have a beautiful day. This hearing is over. 

[Whereupon, at 2 p.m., the subcommittee was adjourned.] 
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